Class Fetch

  • public class Fetch
extends java.lang.Object

    This class was built using the Chrome Remote Dev-Tools A.P.I., which is specified by two JSON-RPC Files. These files were obtained from the Chrome Dev Tools Protocol Git Hub Page, which has a "Tip of Tree" (the latest) API-Specification Page Here: JSON-RPC Protocol Specification.

    These files were converted into this Java-Browser (CDP) Library. The intention is to have them function in a similar fasion to the Node.js Tool known as 'Puppeteer', Microsoft's 'Playwright' and of course the Main-Stay 'Selenium.' The Java-HTML JAR Library merely implements the Java Types & Commands defined by Google's DevTools Protocol.

    🧠 View the Google CDP API:

    A domain for letting clients substitute browser's network layer with client code.

    The top-level description and explanation for this class (this comment, at the top this Java-Doc Page) is repeated, verbatim, across all of the domain classes which comprise Google's CDP API.

    This class is intended to be used with a Browser Instance

    These methods have been tested, to some degree, using Google Chrome. In order to use this class you must start a web-browser instance and make a connection to the browser using a Remote Debugging Port. Google-Corporation is the developer of this API, but any browser which accepts a Remote Debug Port Connection over Web-Sockets.

    Google-Chrome was used during the development process of the classes in this particular package. Lately, it has been asserted Microsoft has switched to using the Chrome Browser-Engine for its Microsoft Edge Internal Code-Base. Therefore, there may some functionality available when running the methods in this class with Microsoft-Edge.

    Check whether the your Web-Browser will allow itself to be driven by the Web-Socket RDP-Port 9223. See the examples available in package Torello.Browser to undertand how to build a PageConn and BrowserConn Web-Socket Connection, and how to build a WebSocketSender instance in order to execute the methods in this class.

    Web-Socket & JSON API:   
    Every one of the methods that reside in this class are designed to do nothing more than:

    1. Accept Parameters from the User, and "Marshall Them" into a Valid JSON-Request
    2. Transmit the Marshalled Request-JSON to a Headless Web-Browser over a Web-Socket Connection
    3. Receive BOTH that Command-Results AND any Browser Event-Firings from the Web-Socket
    4. Parse JSON Method-Results and Browser-Event Firings, and Subsequently Convert them to Standard Java-Types
    5. Report these Method-Results and Browser-Events to the User via a User-Registered, Event-Listener (Events) or a Promise Object (Command Responses / Results)

    Unlike the bulk of the Java HTML JAR Library, there is very little native Java-Code, and very little testing that may be done on any of the classes & methods in this package. The code inside these classes does nothing more than marshall-and-unmarshall Java-Types into Json-Requests (and vice-versa). The Java-Script & Browser modules inside of a Google-Chrome instance are, theoretically, handling these requests, and returning their results (or events) over the Web-Socket Connection.

    It has been asserted (by Google Chrome Developers) that some of these methods are only "partially working" or "experimental".


    Asking Chat-GPT for Help:   
    The LLM otherwise known as "Chat-GPT" does, indeed, have an expert level of knowledge about the "Remote DevTools Protocol". The API that the Chrome DevTools Protocl (CDP) exports is extremely well understood by the LLM, and generally I have found that Chat-GPT understands (by 2 or 3 orders of magnitude) better what my Auto-Generated JSON-Wrappers can do in controlling a Web-Browser than I could ever possibly hope to understand.

    Though not available today, there will soon be an automatically downloadable Token-Stream (AI Embeddings) BUTTON available on my Java-Doc Pages that should hopefully make it extremely easy to post my code-base, RAG Style, to Chat-GPT and other LLM's when 'interogating' them. Presently, because my "Get Token Stream Button" does not exist yet on any of my pages, what you can do is copy-and-paste any Method-Signature from any one of these pages and then ask Chat-GPT to explain what that Browser or Java-Script Function is actually doing. It is very likely to give you some pretty neat answers.

    I have found that every single one of the Domains, Types & Events which are offered by the CDP Protocol (though not documented very well by Google), are perfectly understood by the A.I. LLM - literally to the point where it does know (much better than I ever could) what my own code base actually does!

    Try it out, it's a lot of fun. Note that this package and these classes were originally developed solely to be able to execute the Java-Script that a browser executes when visiting a Web-Site. Complete HTML-Page Content can be scraped (using the HTML Data-Scraping Tools in Java-HTML) off of Web-Sites that have dynamic / Java-Script Generated Content.


    Conspicuous Boxed-Types Usage:
    You may notice that there are many methods that have parameters which accept, for instance, an Integer, instead of a primitive int. Just to remind the readiner, in Java Programs a Boxed Type is a standard Java-Primitive which has been converted into an Object-Reference. The use of Boxed-Types in this code base is an easy-and-fast-way to allow for the concept of "Optional Parameters" or "Optional Field Value."

    Whenever you see a method that accepts an Integer, the reason for this Parameter-Type choice is actually to allow a user to pass 'null' to it. This is a simple way to ELIDE passing any value at all to parameters which Google-Chrome would otherwise assert are "Optional." Whenever you pass 'null' to a Boxed-Types in this class, the Json-Processor will simply eliminate that Object-Property from the command altogether; and the browser will simply not receive any value for that parameter when that command is invoked.

    The Java Language Specification does not have an easy or well defined means of accepting optional method parameters; so Boxed-Types and 'null' are utilized here. Note that 'null' may be passed to any Command Method-Parameter that is listed as Optional on the Java-Doc Page description for that parameter.



    Stateless Class:
    This class neither contains any program-state, nor can it be instantiated. The @StaticFunctional Annotation may also be called 'The Spaghetti Report'. Static-Functional classes are, essentially, C-Styled Files, without any constructors or non-static member fields. It is a concept very similar to the Java-Bean's @Stateless Annotation.

    • 1 Constructor(s), 1 declared private, zero-argument constructor
    • 12 Method(s), 12 declared static
    • 2 Field(s), 2 declared static, 2 declared final


    • Nested Class Summary

       
      Type Nested Classes: Types / Classes that Are Used & Exported by this Domain
      Modifier and Type Class Description
      static class  Fetch.AuthChallenge
      Authorization challenge for HTTP status code 401 or 407.
      static class  Fetch.AuthChallengeResponse
      Response to an AuthChallenge.
      static class  Fetch.HeaderEntry
      Response HTTP header entry
      static class  Fetch.RequestPattern
      [No Description Provided by Google]
       
      Event Nested Classes: Browser Events, as Java Inner Classes, Which are Fired by this Domain
      Modifier and Type Class Description
      static class  Fetch.authRequired
      Issued when the domain is enabled with handleAuthRequests set to true.
      static class  Fetch.requestPaused
      Issued when the domain is enabled and the request URL matches the specified filter.
       
      Command-Returns Nested Classes: Domain-Commands with Multiple Return-Values, and a Dedicated Inner-Class
      Modifier and Type Class Description
      static class  Fetch.getResponseBody$$RET
      Causes the body of the response to be received from the server and returned as a single string.

    • Field Summary

       
      Enumerated Strings: Like Java 'enum' Types, but Converted to Read-Only String-Lists
      Modifier and Type Field Description
      static ReadOnlyList<String> RequestStage
      Stages of the request to handle.
       
      Eliminated Types: Removed CDP Types which Have Been Re-Mapped to Basic Java String Constants
      Modifier and Type Field Description
      static String RequestId
      Unique request identifier.

    • Method Summary

       
      Fetch Domain Commands
      Script Returns Modifier and Type Method
      Void static Script<> continueRequest​(String requestId, String url, String method, String postData, Fetch.HeaderEntry[] headers, Boolean interceptResponse)
      Continues the request, optionally modifying some of its parameters.
      Void static Script<> continueResponse​(String requestId, Integer responseCode, String responsePhrase, Fetch.HeaderEntry[] responseHeaders, String binaryResponseHeaders)
      Continues loading of the paused response, optionally modifying the response headers.
      Void static Script<> continueWithAuth​(String requestId, Fetch.AuthChallengeResponse authChallengeResponse)
      Continues a request supplying authChallengeResponse following authRequired event.
      Void static Script<> disable()
      Disables the fetch domain.
      Void static Script<> enable​(Fetch.RequestPattern[] patterns, Boolean handleAuthRequests)
      Enables issuing of requestPaused events.
      Void static Script<> failRequest​(String requestId, String errorReason)
      Causes the request to fail with specified reason.
      Void static Script<> fulfillRequest​(String requestId, int responseCode, Fetch.HeaderEntry[] responseHeaders, String binaryResponseHeaders, String body, String responsePhrase)
      Provides response to the request.
      Fetch.getResponseBody$$RET static Script<> getResponseBody​(String requestId)
      Causes the body of the response to be received from the server and returned as a single string.
      String static Script<> takeResponseBodyAsStream​(String requestId)
      Returns a handle to the stream representing the response body.
       
      Fetch Domain CommandBuilder Methods
      Modifier and Type Method Description
      static CommandBuilder
      <Void>      		 continueRequest()
      Creates a buider for conveniently assigning parameters to this method.
      static CommandBuilder
      <Void>      		 continueResponse()
      Creates a buider for conveniently assigning parameters to this method.
      static CommandBuilder
      <Void>      		 fulfillRequest()
      Creates a buider for conveniently assigning parameters to this method.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Field Detail

      • RequestId

        🡇  ⇈  ⮫ 🗕  🗗  🗖 
        public static final java.lang.String RequestId
        Unique request identifier. Note that this does not identify individual HTTP requests that are part of a network request.

        The Type RequestId has been eliminated, because it is a direct mapping to a basic Java-Type; it has no additional fields, or other distinguishing properties. Instead, this CDP defined type has been relegated to a simple String Constant, for documentation & reference purposes only.

        The code which is generated which employs this type replaces its use with the Standard Java-Type: String

        Eliminated Type
        See Also:
        Constant Field Values

      • RequestStage

        🡅  🡇  ⇈  ⮫ 🗕  🗗  🗖 
        public static final ReadOnlyList<java.lang.String> RequestStage
        Stages of the request to handle. Request will intercept before the request is sent. Response will intercept after the response is received (but before response body is received).

        String-Enumeration Type

    • Method Detail

      • continueRequest

        🡅  🡇  ⇈  ⮫ 🗕  🗗  🗖 
        public static Script<java.lang.Void> continueRequest​
            (java.lang.String requestId,
             java.lang.String url,
             java.lang.String method,
             java.lang.String postData,
             Fetch.HeaderEntry[] headers,
             java.lang.Boolean interceptResponse)
        Continues the request, optionally modifying some of its parameters.

        👍 Because of the sheer number of input parameters to this method, there is a a CommandBuilder variant to this method which may be invoked instead.

        Please View: continueRequest()
        Parameters:
        requestId - An id the client received in requestPaused event.
        url - If set, the request url will be modified in a way that's not observable by page.
        OPTIONAL
        method - If set, the request method is overridden.
        OPTIONAL
        postData - If set, overrides the post data in the request. (Encoded as a base64 string when passed over JSON)
        OPTIONAL
        headers - If set, overrides the request headers. Note that the overrides do not extend to subsequent redirect hops, if a redirect happens. Another override may be applied to a different request produced by a redirect.
        OPTIONAL
        interceptResponse - If set, overrides response interception behavior for this request.
        OPTIONALEXPERIMENTAL
        Returns:
        An instance of Script<Void>

        This Script instance must be executed before the browser receives the invocation-request.

        This Browser-Function does not have a return-value. You may choose to await the Promise<Void> to ensure that the Browser Function has run to completion.

      • continueResponse

        🡅  🡇  ⇈  ⮫ 🗕  🗗  🗖 
        public static Script<java.lang.Void> continueResponse​
            (java.lang.String requestId,
             java.lang.Integer responseCode,
             java.lang.String responsePhrase,
             Fetch.HeaderEntry[] responseHeaders,
             java.lang.String binaryResponseHeaders)
        Continues loading of the paused response, optionally modifying the response headers. If either responseCode or headers are modified, all of them must be present.
        EXPERIMENTAL

        👍 Because of the sheer number of input parameters to this method, there is a a CommandBuilder variant to this method which may be invoked instead.

        Please View: continueResponse()
        Parameters:
        requestId - An id the client received in requestPaused event.
        responseCode - An HTTP response code. If absent, original response code will be used.
        OPTIONAL
        responsePhrase - A textual representation of responseCode. If absent, a standard phrase matching responseCode is used.
        OPTIONAL
        responseHeaders - Response headers. If absent, original response headers will be used.
        OPTIONAL
        binaryResponseHeaders - Alternative way of specifying response headers as a \0-separated series of name: value pairs. Prefer the above method unless you need to represent some non-UTF8 values that can't be transmitted over the protocol as text. (Encoded as a base64 string when passed over JSON)
        OPTIONAL
        Returns:
        An instance of Script<Void>

        This Script instance must be executed before the browser receives the invocation-request.

        This Browser-Function does not have a return-value. You may choose to await the Promise<Void> to ensure that the Browser Function has run to completion.

      • continueWithAuth

        🡅  🡇  ⇈  ⮫ 🗕  🗗  🗖 
        public static Script<java.lang.Void> continueWithAuth​
            (java.lang.String requestId,
             Fetch.AuthChallengeResponse authChallengeResponse)
        Continues a request supplying authChallengeResponse following authRequired event.
        Parameters:
        requestId - An id the client received in authRequired event.
        authChallengeResponse - Response to with an authChallenge.
        Returns:
        An instance of Script<Void>

        This Script instance must be executed before the browser receives the invocation-request.

        This Browser-Function does not have a return-value. You may choose to await the Promise<Void> to ensure that the Browser Function has run to completion.

      • disable

        🡅  🡇  ⇈  ⮫ 🗕  🗗  🗖 
        public static Script<java.lang.Void> disable()
        Disables the fetch domain.
        Returns:
        An instance of Script<Void>

        This Script instance must be executed before the browser receives the invocation-request.

        This Browser-Function does not have a return-value. You may choose to await the Promise<Void> to ensure that the Browser Function has run to completion.

      • enable

        🡅  🡇  ⇈  ⮫ 🗕  🗗  🗖 
        public static Script<java.lang.Void> enable​
            (Fetch.RequestPattern[] patterns,
             java.lang.Boolean handleAuthRequests)
        Enables issuing of requestPaused events. A request will be paused until client calls one of failRequest, fulfillRequest or continueRequest/continueWithAuth.
        Parameters:
        patterns - If specified, only requests matching any of these patterns will produce fetchRequested event and will be paused until clients response. If not set, all requests will be affected.
        OPTIONAL
        handleAuthRequests - If true, authRequired events will be issued and requests will be paused expecting a call to continueWithAuth.
        OPTIONAL
        Returns:
        An instance of Script<Void>

        This Script instance must be executed before the browser receives the invocation-request.

        This Browser-Function does not have a return-value. You may choose to await the Promise<Void> to ensure that the Browser Function has run to completion.

      • failRequest

        🡅  🡇  ⇈  ⮫ 🗕  🗗  🗖 
        public static Script<java.lang.Void> failRequest​
            (java.lang.String requestId,
             java.lang.String errorReason)
        Causes the request to fail with specified reason.
        Parameters:
        requestId - An id the client received in requestPaused event.
        errorReason - Causes the request to fail with the given reason.
        Returns:
        An instance of Script<Void>

        This Script instance must be executed before the browser receives the invocation-request.

        This Browser-Function does not have a return-value. You may choose to await the Promise<Void> to ensure that the Browser Function has run to completion.

      • fulfillRequest

        🡅  🡇  ⇈  ⮫ 🗕  🗗  🗖 
        public static Script<java.lang.Void> fulfillRequest​
            (java.lang.String requestId,
             int responseCode,
             Fetch.HeaderEntry[] responseHeaders,
             java.lang.String binaryResponseHeaders,
             java.lang.String body,
             java.lang.String responsePhrase)
        Provides response to the request.

        👍 Because of the sheer number of input parameters to this method, there is a a CommandBuilder variant to this method which may be invoked instead.

        Please View: fulfillRequest()
        Parameters:
        requestId - An id the client received in requestPaused event.
        responseCode - An HTTP response code.
        responseHeaders - Response headers.
        OPTIONAL
        binaryResponseHeaders - Alternative way of specifying response headers as a \0-separated series of name: value pairs. Prefer the above method unless you need to represent some non-UTF8 values that can't be transmitted over the protocol as text. (Encoded as a base64 string when passed over JSON)
        OPTIONAL
        body - A response body. If absent, original response body will be used if the request is intercepted at the response stage and empty body will be used if the request is intercepted at the request stage. (Encoded as a base64 string when passed over JSON)
        OPTIONAL
        responsePhrase - A textual representation of responseCode. If absent, a standard phrase matching responseCode is used.
        OPTIONAL
        Returns:
        An instance of Script<Void>

        This Script instance must be executed before the browser receives the invocation-request.

        This Browser-Function does not have a return-value. You may choose to await the Promise<Void> to ensure that the Browser Function has run to completion.

      • getResponseBody

        🡅  🡇  ⇈  ⮫ 🗕  🗗  🗖 
        public static Script<Fetch.getResponseBody$$RET> getResponseBody​
            (java.lang.String requestId)
        Causes the body of the response to be received from the server and returned as a single string. May only be issued for a request that is paused in the Response stage and is mutually exclusive with takeResponseBodyForInterceptionAsStream. Calling other methods that affect the request or disabling fetch domain before body is received results in an undefined behavior. Note that the response body is not available for redirects. Requests paused in the _redirect received_ state may be differentiated by responseCode and presence of location response header, see comments to requestPaused for details.
        Parameters:
        requestId - Identifier for the intercepted request to get body for.
        Returns:
        An instance of Script<Fetch.getResponseBody$$RET>

        This script may be executed, using Script.exec, and afterwards, a Promise <Fetch.getResponseBody$$RET> will be returned

        Finally, the Promise may be awaited, using Promise.await(), and the returned result of this Browser Function may be retrieved.

        This Browser Function's Promise returns:Fetch.getResponseBody$$RET A dedicated return type implies that the browser may return more than 1 datum

      • takeResponseBodyAsStream

        🡅  🡇  ⇈  ⮫ 🗕  🗗  🗖 
        public static Script<java.lang.String> takeResponseBodyAsStream​
            (java.lang.String requestId)
        Returns a handle to the stream representing the response body. The request must be paused in the HeadersReceived stage. Note that after this command the request can't be continued as is -- client either needs to cancel it or to provide the response body. The stream only supports sequential read, IO.read will fail if the position is specified. This method is mutually exclusive with getResponseBody. Calling other methods that affect the request or disabling fetch domain before body is received results in an undefined behavior.
        Parameters:
        requestId - -
        Returns:
        An instance of Script<String>

        This script may be executed, using Script.exec, and afterwards, a Promise <String> will be returned

        Finally, the Promise may be awaited, using Promise.await(), and the returned result of this Browser Function may be retrieved.

        This Browser Function's Promise returns: String (stream)