Package Torello.Java
Class LFEC.GCSSB
- java.lang.Object
-
- Torello.Java.LFEC.GCSSB
-
- Enclosing class:
- LFEC
public static class LFEC.GCSSB extends java.lang.Object
The Google Cloud Server Storage Bucket extension of "Load File Exception Catch" does the work thatLFEC
does, but for GCS Storage Buckets, rather than operating system files.
The followingpublic static (inner) class
provides the exact same set of features thatclass LFEC
provides - namely data-file loading with failing print's and system halts - but for files that are stored on the Google Cloud Server Storage Bucket Infrastructure.
The set classes of classes and methods that are 'exported' by the HTML search and scrape package, and the other packages in this Torello Jar File are neither dependent upon, nor do they mandate or require use of Google Cloud Services, the 'Cloud Shell' development-environment that google provides (for free), nor any of the Storage-Buckets or storage-systems that Google Corporation operates. Literally all of the development work for package HTML was done on Google's Cloud Shell, but hopefully it will all run - in the spirit of Java's write-once, run-anywhere platform proclamations - anywhere that the JAR files are loaded. If you are planning to utilize Google's Cloud Server to either develop your code, or to host your web-site, the method & classes that link to its servers in this class-infrastructure might seem invaluable. The"Storage Buckets" system,
for instance, which they offer are pretty cheap - I have about 20 to 30 Gigabytes there right now, and have paid probably $1.50 per month to them...
USING GCS: In order to use the Google Cloud Services platform, you will have to do one of the following two means of communicating commands and instructions to their servers:- Make
json
calls to the appropriate server-names using whatever "usual http-connect methods" that you employ, and parse thejson response
object. Parsing json-response objects can be really easy if you know how to use regular-expressions, but the "Jackson" library also does this for free. Download a copy of the "Jackson" java jar-file, and read the "Jackson JSON Interpreter JavaDoc's" on the internet. (Type:Jackson Java JSON parse
at a "google search prompt"). - ... or ... spend however long it takes to read, figure-out, understand (or interpret) Google's GCS Java-Library and write java-code using their JSON-free java-library making direct calls to Google's GCS servers.
HERE, JAVA: The methods in this class do not use the "JSON" version of communicating with Google-Servers, instead, they use the java class-libraries that Google Exports. I do not know which the "official version" of it's jar-file is, but you may download an "unofficial-google version" from my web-site, which does work. I have not tampered with it, or altered the classes in any-way, other than to remove some of the complicated "extras" that were added.
DOWNLOAD HERE: This is a download of file "GCS.jar" which has the classes and methods needed to access the translate API, the Storage Bucket API, and the Authorization classes"o-auth 2.0"
. If you make any calls to the methods here that facilitate access to the Storage Buckets, you will need to include some version of Google's GCS jar file. This is the jar-file I created, if you don't trust Torello.Directory, please don't use it:
JAR DOWNLOAD-LINK: http://developer.torello.directory/ralphsReference/jars/GCS.jar
DOWNLOAD-LINK 2: GCS.jar
I am not really able to make any proclamations for or against about the contents of this particular jar-file. I am, personally, not a fan of the Java-Enabled versions of its services that are provided by Google, primarily because I have almost no idea how to use them (Google-Java is very poorly documented, and both the classes and the methods they offer are not well thought as API's - but rather they just expose their own internal stuff). One may usually figure GCS Services using the JSON, and just go with making JSON calls to-and-from GCS. Anyway, in the above java-jar file many (but not all) of the classes necessary to avoid falling back on JSON calls to GCS are available. Using Google's jar-files, it is possible to make "plain old java method calls" instead of using JSON. Most Importantly: You do not need this jar file included in your classpath for any of the methods or classes in this scrape package at all - unless - you have a GCS account, and want to use their Storage Buckets. If so, the two methods inclass 'Programming'
that use the term:'GCSSB'
an acronym that means"Google Cloud Server Storage Buckets"
FINALLY: The following code snippet should explain how to provide an "o-auth-2.0" to obtain an instance of theclass Storage
needed to begin communicating with Google Cloud Server Storage Buckets:
import com.google.cloud.storage.*; import com.google.auth.oauth2.*; ... String PROJECT_ID = "The name you gave your project in which your storage buckets reside"; String PATH_TO_JSON_KEY = "The JSON key generated when you create a 'Service Account'."; Storage storage = StorageOptions .newBuilder() .setProjectId(PROJECT_ID) .setCredentials(GoogleCredentials.fromStream(new FileInputStream(PATH_TO_JSON_KEY))) .build() .getService();
SPECIAL NOTE: I have never needed myGoogleCredentials "o auth 2.0"
key until I started using the google-java jar-libraries for connecting withStorage Buckets
directly.- When writing to my web-domain, using the command line program
GSUTIL
was usually much easier and eliminated the strict dependency on Google's web-hosting platform that would be mandated if using their Java-Libraries. - When translating Chinese Government web-publications using the Translate API, a Google "API Key" usually sufficed. An "API Key" is a 30 to 40 characters string that identifies the billable account to the server. Just don't share it on the web, and it will work fine.
- Even the Vision / OCR (Optical Character Recognition) API's seemed to work without oauth2.0. I did have to make HTTP connections and interpret JSON, rather than using bona-fide java method invocations, but it was all fine. If you are going to use java-methods with the storage-buckets, get an "o-auth-2.0" key, and save the JSON file to your file-system. Keep that key private, and the previous code is how to "login to google" from java and start saving storing files to the cloud using their java jar-files.
It should be interesting to note, that often, this method invocation works - and I don't actually know why, and cannot explain it. This leaves out the authentication portion from the Storage object, and sometimes still allows file access to the files in your storage-buckets. You may play around with it:
import import com.google.cloud.storage.*; ... private static final Storage storage = StorageOptions.getDefaultInstance().getService();
NOTE: The easiest way to use this class is to make sure to import the following line in your import package's (and classes!) section of your class-file. When you include the complete path name to a static-inner class in a Java'import package'
section of your java-class-definition ('.java'
file), then through-out the remainder of your code, you may make calls to this class directly, without being required to type/enter the entire class-name on each and every line.
// When this line is included, Using class' static method is done as follows: // GCSSB.loadFileToString(storage, "mybucket", "myFile.txt"); // If this import-statement is not used, then (as for all static/internal classes), you must use // the entire class-name: // LFEC.GCSSB.loadFileToString( ... ); import Torello.Java.LFEC.GCSSB;
Hi-Lited Source-Code:- View Here: Torello/Java/LFEC.java
- Open New Browser-Tab: Torello/Java/LFEC.java
Stateless Class:This class neither contains any program-state, nor can it be instantiated. The@StaticFunctional
Annotation may also be called 'The Spaghetti Report'.Static-Functional
classes are, essentially, C-Styled Files, without any constructors or non-static member fields. It is a concept very similar to the Java-Bean's@Stateless
Annotation.
- 1 Constructor(s), 1 declared private, zero-argument constructor
- 5 Method(s), 5 declared static
- 0 Field(s)
-
-
Method Summary
Read Files from Google Cloud Platform Storage Buckets Modifier and Type Method static String
loadFileToString(com.google.cloud.storage.Storage storage, String bucket, String completeFileName)
static Vector<String>
loadFileToVector(com.google.cloud.storage.Storage storage, String bucket, String completeFileName, boolean includeNewLine)
static <T> T
readObjectFromFile(com.google.cloud.storage.Storage storage, String bucket, String completeFileName, boolean zip, Class<T> returnClass)
Write Files to Google Cloud Platform Storage Buckets Modifier and Type Method static void
writeFile(CharSequence fileAsStr, com.google.cloud.storage.Storage storage, String bucket, String completeFileName, boolean ASCIIorUTF8)
static void
writeObjectToFile(Object o, com.google.cloud.storage.Storage storage, String bucket, String completeFileName, boolean zip)
-
-
-
Method Detail
-
readObjectFromFile
public static <T> T readObjectFromFile (com.google.cloud.storage.Storage storage, java.lang.String bucket, java.lang.String completeFileName, boolean zip, java.lang.Class<T> returnClass)
This will read a Java Serializedjava.lang.Object
from a location in a Google Cloud ServerStorage Bucket
.
NOTE: This uses Java's variable-type parameter syntax. To easily provide a value for parameter'returnClass'
, type the name of the java'class'
whose type is needed as a return-type, followed by the word'class'
.
For example, to read ajava.util.Vector
from a data-file, pass the value'Vector.class'
to this parameter. It is a 'little-known' fact that each and every instance ofjava.lang.Object
has aClass class
field inside. (Now, boys and girls, don't start jumping up and down just yet!)- Type Parameters:
T
- This type parameter is simply provided for convenience, to allow the user to specify the return class, without having to cast the object and suppress warnings, or catch exceptions.- Parameters:
storage
- This must be an instance of Google Cloud Server'sclass Storage
. The description at the top of this class should elucidate how to obtain such anObject
-instance. If the explanation is not very clear, or if it is not working (any more), please just go to a google-search-bar and look for information about theGoogle Cloud Server Storage Buckets Java API, class 'Storage'
. API's have been known to change, once in a while.bucket
- Thebucket
name of thebucket
from a Google Cloud Server account.completeFileName
- ThisString
-parameter needs to be the completeString
representation of the directory-name plus the file-name of the location where the java serializedObject
file was saved, or will be saved.
GCS Storage-Buckets: The bucket-name as-a-string should **not** be included as a part of thisString
-parameter. However, both the file-name, and the directory-name where this file is residing must be present.zip
- When this parameter is TRUE the serializedobject
will be run through Java'sjava.util.zip.GZIPInputStream
andGZIPOutputStream
when reading/writing the serialized-Object.
If this parameter is FALSE, GZIP Compression will not be used when serializing or de-serializing theObject.
returnClass
- This is the type expected to be found by Java in the SerializedObject
Data-File. If anObject
is read from this location, but it does not have the type indicated by this parameter, the program will also halt, and an explanatory exception message will be printed to the console/terminal.- Returns:
- A de-serialized java
java.lang.Object
that has been read from a GCSStorage Bucket
, and cast to the type denoted by parameter'returnClass'
. - See Also:
FileRW.readObjectFromFile(String, boolean)
,FileRW.readObjectFromFileNOCNFE(String, boolean)
,LFEC.readObjectFromFile(String, boolean, Class)
,LFEC.ERROR_EXIT(String)
- Code:
- Exact Method Body:
try { // Read Storage Bucket Data into a byte[] array byte[] bArr = storage.get(bucket, completeFileName).getContent(); // Build an Input Stream, using that byte[] array as input ByteArrayInputStream bis = new ByteArrayInputStream(bArr); // Build an Object Input Stream, using the byte-array input-stream as input ObjectInputStream ois = zip ? new ObjectInputStream(new GZIPInputStream(bis)) : new ObjectInputStream(bis); // Use Java's Object Serialization method to read the Object Object ret = ois.readObject(); if (! returnClass.isInstance(ret)) ERROR_EXIT( "Serialized Object read from GCS Storage Bucket: " + bucket + "\n" + "And file-name: " + completeFileName + "\n" + "Using expected (" + (zip ? "zip-compression" : "no-compression") + ")\n" + "Didn't have an object with class-name: " + returnClass + "\n" + "But rather with className: " + ret.getClass().getName() ); ois.close(); bis.close(); return returnClass.cast(ret); } catch (Throwable t) { ERROR_EXIT( t, "Serialized Object read from GCS Storage Bucket: " + bucket + "\n" + "And file-name: " + completeFileName + "\n" + "Using expected (" + (zip ? "zip-compression" : "no-compression") + ")\n" + "And Expected class-name: " + returnClass + "\n" ); throw new UnreachableError(); // Cannot reach this statement }
-
loadFileToString
public static java.lang.String loadFileToString (com.google.cloud.storage.Storage storage, java.lang.String bucket, java.lang.String completeFileName)
This merely loads a text-file from Google's Storage Bucket infrastructure into aString.
Make sure to check that the file you are loading does indeed have text-content.- Parameters:
storage
- This must be an instance of Google Cloud Server'sclass Storage
. The description at the top of this class should elucidate how to obtain such anObject
-instance. If the explanation is not very clear, or if it is not working (any more), please just go to a google-search-bar and look for information about theGoogle Cloud Server Storage Buckets Java API, class 'Storage'
. API's have been known to change, once in a while.bucket
- Thebucket
name of thebucket
from a Google Cloud Server account.completeFileName
- ThisString
-parameter needs to be the completeString
representation of the directory-name plus the file-name of the location where the java serializedObject
file was saved, or will be saved.
GCS Storage-Buckets: The bucket-name as-a-string should **not** be included as a part of thisString
-parameter. However, both the file-name, and the directory-name where this file is residing must be present.- Returns:
- The text file on Google Cloud Server's Storage Bucket file/directory returned as
a
java.lang.String
- Code:
- Exact Method Body:
return new String(storage.get(bucket, completeFileName).getContent());
-
loadFileToVector
public static java.util.Vector<java.lang.String> loadFileToVector (com.google.cloud.storage.Storage storage, java.lang.String bucket, java.lang.String completeFileName, boolean includeNewLine)
This merely loads a text-file from Google's Storage Bucket infrastructure into aString
. Make sure to check that the file you are loading does indeed have text-content.- Parameters:
storage
- This must be an instance of Google Cloud Server'sclass Storage
. The description at the top of this class should elucidate how to obtain such anObject
-instance. If the explanation is not very clear, or if it is not working (any more), please just go to a google-search-bar and look for information about theGoogle Cloud Server Storage Buckets Java API, class 'Storage'
. API's have been known to change, once in a while.bucket
- Thebucket
name of thebucket
from a Google Cloud Server account.completeFileName
- ThisString
-parameter needs to be the completeString
representation of the directory-name plus the file-name of the location where the java serializedObject
file was saved, or will be saved.
GCS Storage-Buckets: The bucket-name as-a-string should **not** be included as a part of thisString
-parameter. However, both the file-name, and the directory-name where this file is residing must be present.includeNewLine
- This tells the method to include, or not-include, a'\n'
(newline) character to eachString
.- Returns:
- The text file on Google Cloud Server's
Storage Bucket
file/directory stuff as aVector
ofString's
. - See Also:
loadFileToString(Storage, String, String)
- Code:
- Exact Method Body:
String s = loadFileToString(storage, bucket, completeFileName); Vector<String> ret = new Vector<>(); int pos = 0; int delta = includeNewLine ? 1 : 0; int lastPos = 0; while ((pos = s.indexOf('\n')) != -1) { ret.add(s.substring(lastPos, pos + delta)); lastPos = pos + 1; } if (lastPos < s.length()) ret.add(s.substring(lastPos)); return ret;
-
writeFile
public static void writeFile(java.lang.CharSequence fileAsStr, com.google.cloud.storage.Storage storage, java.lang.String bucket, java.lang.String completeFileName, boolean ASCIIorUTF8)
This will write the contents of a java'CharSequence'
- includesString, StringBuffer & StringBuilder
to a file on Google Cloud Server's storage bucket system.- Parameters:
storage
- This must be an instance of Google Cloud Server'sclass Storage
. The description at the top of this class should elucidate how to obtain such anObject
-instance. If the explanation is not very clear, or if it is not working (any more), please just go to a google-search-bar and look for information about theGoogle Cloud Server Storage Buckets Java API, class 'Storage'
. API's have been known to change, once in a while.bucket
- Thebucket
name of thebucket
from a Google Cloud Server account.completeFileName
- ThisString
-parameter needs to be the completeString
representation of the directory-name plus the file-name of the location where the java serializedObject
file was saved, or will be saved.
GCS Storage-Buckets: The bucket-name as-a-string should **not** be included as a part of thisString
-parameter. However, both the file-name, and the directory-name where this file is residing must be present.ASCIIorUTF8
- When writing javaString's
the file-system, it is generally not to important to worry about whether java has stored an'ASCII'
encodedString
, or aString
encoded using'UTF-8'
. Most foreign-language news-sites require the latter ('UTF-8'
), but any site that is strictly English can get by with plain old ASCII.
IMPORTANT: When this boolean is TRUE, this method will attempt to presume the character-sequence you have passed is in ASCII, and write it that way. When this boolean is set to FALSE, this method will attempt to write theString
ofbyte's
as a'UTF-8'
encoded character-set.
ALSO: I have not made any allowance for Unicode or Unicode little endian, because I have never used them with either the Chinese or Spanish sites I scrape. UTF-8 has been the only other character set I encounter.- Code:
- Exact Method Body:
BlobInfo blobInfo = BlobInfo.newBuilder (BlobId.of(bucket, completeFileName)).setContentType("text/plain").build(); byte[] file = ASCIIorUTF8 ? fileAsStr.toString().getBytes() : fileAsStr.toString().getBytes(java.nio.charset.Charset.forName("UTF-8")); Blob blob = storage.create(blobInfo, file);
-
writeObjectToFile
public static void writeObjectToFile (java.lang.Object o, com.google.cloud.storage.Storage storage, java.lang.String bucket, java.lang.String completeFileName, boolean zip) throws java.io.IOException
This will write a JavaSerializable Object
to a location in a Google Cloud Server Storage Bucket.- Parameters:
storage
- This must be an instance of Google Cloud Server'sclass Storage
. The description at the top of this class should elucidate how to obtain such anObject
-instance. If the explanation is not very clear, or if it is not working (any more), please just go to a google-search-bar and look for information about theGoogle Cloud Server Storage Buckets Java API, class 'Storage'
. API's have been known to change, once in a while.o
- This may be anySerializable Java Object
.Serializable Java Objects
are ones which implement theinterface java.io.Serializable
.bucket
- Thebucket
name of thebucket
from a Google Cloud Server account.completeFileName
- ThisString
-parameter needs to be the completeString
representation of the directory-name plus the file-name of the location where the java serializedObject
file was saved, or will be saved.
GCS Storage-Buckets: The bucket-name as-a-string should **not** be included as a part of thisString
-parameter. However, both the file-name, and the directory-name where this file is residing must be present.zip
- When this parameter is TRUE the serializedobject
will be run through Java'sjava.util.zip.GZIPInputStream
andGZIPOutputStream
when reading/writing the serialized-Object.
If this parameter is FALSE, GZIP Compression will not be used when serializing or de-serializing theObject.
- Throws:
java.io.IOException
- Code:
- Exact Method Body:
// Retrieves a file-name object using a GCS BUCKET-NAME, and the FILE-NAME (in // the bucket) BlobId blobId = BlobId.of(bucket, completeFileName); // This BlobInfo is GCS version of "java.io.File". It points to a specific file // inside a GCS Bucket (which was specified earlier) BlobInfo blobInfo = BlobInfo.newBuilder(blobId) /* .setContentType("text/plain") */.build(); // This will save the Serialized Object Data to a Stream (and eventually an array) ByteArrayOutputStream baos = new ByteArrayOutputStream(); // This stream writes serialized Java-Objects to the Storage Bucket ObjectOutputStream oos = zip ? new ObjectOutputStream(new GZIPOutputStream(baos)) : new ObjectOutputStream(baos); oos.writeObject(o); oos.flush(); baos.flush(); oos.close(); // Convert that BAOS to a Byte-Array byte[] bArr = baos.toByteArray(); // Write the BYTE-ARRAY to the GCS Bucket and file using the "BlobInfo" that was built // a few lines ago. Blob blob = storage.create(blobInfo, bArr);
-
-