Loop Google Storage (Cloud Connectivity)
SynopsisThis operator loops over all files in the specified bucket/folder from the Google Cloud Storage.
After you have configured your Google account, you can process all Google Storage files within the selected folder.
Be aware that the operator cannot read the file as example set. For this reason, you must connect the file input in the inner process of this operator to another appropriate operator to process the file. For example, if you want to load Excel files from your Google Storage folder, you must connect the file input in the inner process with the Read Excel operator.
- in (IOObject)
Optional input data which is delivered to the inner process.
- connection (Connection)
This input port expects a Connection object if any. See the parameter connection entry for more information.
- out (IOObject)
Output data of the inner process.
- connection (Connection)
This output port delivers the Connection object from the input port. If the input port is not connected the port delivers nothing.
- connection_source This parameter indicates how the connection should be specified. It gives you two options, predefined and repository. The parameter is not visible if the connection input port is connected. Range: selection
- connection_entry This parameter is only available when the connection source parameter is set to repository. This parameter is used to specify a repository location that represents a connection entry. The connection can also be provided using the connection input port. Range: string
- connection This parameter is only available when the connection source parameter is set to predefined. The connection details for the Google Storage connection can be specified here. If you have already configured a Google Storage connection, you can select it from the drop-down list. If you have not configured a Google Storage connection yet, select the icon to the right of the drop-down list. Create a new Google Storage connection in the Manage connections box. The access token / private key and project ID are required. Range: configurable
- folder Provide the name of the Google Storage folder over which you want to loop. Range: selection
- filter Optional filter via a regular expression which is used to exclude files from looping over them, e.g. 'a.*b' for all files starting with 'a' and ending with 'b'. Ignored if empty. Range: string
- filtered_stringIndicates which part of the file name is matched against the filter expression.
- file_name: Filtered on the name, e.g. 'myfile.txt'
- full_path: Filtered on the full path, e.g. 'mybucket/myfolder/myfile.txt'
- parent_path: Filtered on the parent folder, e.g. 'myfolder/'
- file_name_macro The name of the macro which will contain the name of the current file for each file the loop iterates over, e.g. 'myfile.txt' Range: string
- file_path_macro The name of the macro which will contain the full path of the current file for each file the loop iterates over, e.g. 'mybucket/myfolder/myfile.txt' Range: string
- parent_path_macro The name of the macro which will contain the parent folder of the current file for each file the loop iterates over, e.g. 'myfolder/' Range: string
- recursive If selected, the loop will also iterate over all files in all subfolders of the selected folder. Otherwise, it will only iterate over the files in the selected folder. Range: boolean