Search in sources :

Example 1 with ImplicitColumns

use of org.apache.drill.exec.store.ImplicitColumnUtils.ImplicitColumns in project drill by apache.

the class HttpXMLBatchReader method open.

@Override
public boolean open(SchemaNegotiator negotiator) {
    HttpUrl url = buildUrl();
    // Result set loader setup
    String tempDirPath = negotiator.drillConfig().getString(ExecConstants.DRILL_TMP_DIR);
    // Create user-friendly error context
    CustomErrorContext errorContext = new ChildErrorContext(negotiator.parentErrorContext()) {

        @Override
        public void addContext(UserException.Builder builder) {
            super.addContext(builder);
            builder.addContext("URL", url.toString());
        }
    };
    negotiator.setErrorContext(errorContext);
    // Http client setup
    SimpleHttp http = SimpleHttp.builder().scanDefn(subScan).url(url).tempDir(new File(tempDirPath)).paginator(paginator).proxyConfig(proxySettings(negotiator.drillConfig(), url)).errorContext(errorContext).build();
    // Get the input stream
    inStream = http.getInputStream();
    // Initialize the XMLReader the reader
    try {
        xmlReader = new XMLReader(inStream, dataLevel, maxRecords);
        resultLoader = negotiator.build();
        if (implicitColumnsAreProjected()) {
            implicitColumns = new ImplicitColumns(resultLoader.writer());
            buildImplicitColumns();
            populateImplicitFieldMap(http);
        }
        RowSetLoader rootRowWriter = resultLoader.writer();
        xmlReader.open(rootRowWriter, errorContext);
        xmlReader.implicitFields(implicitColumns);
    } catch (XMLStreamException e) {
        throw UserException.dataReadError(e).message("Error opening XML stream: " + e.getMessage()).addContext(errorContext).build(logger);
    }
    return true;
}
Also used : SimpleHttp(org.apache.drill.exec.store.http.util.SimpleHttp) XMLStreamException(javax.xml.stream.XMLStreamException) ChildErrorContext(org.apache.drill.common.exceptions.ChildErrorContext) RowSetLoader(org.apache.drill.exec.physical.resultSet.RowSetLoader) File(java.io.File) HttpUrl(okhttp3.HttpUrl) XMLReader(org.apache.drill.exec.store.xml.XMLReader) ImplicitColumns(org.apache.drill.exec.store.ImplicitColumnUtils.ImplicitColumns) CustomErrorContext(org.apache.drill.common.exceptions.CustomErrorContext)

Example 2 with ImplicitColumns

use of org.apache.drill.exec.store.ImplicitColumnUtils.ImplicitColumns in project drill by apache.

the class HttpBatchReader method open.

@Override
public boolean open(SchemaNegotiator negotiator) {
    // Result set loader setup
    String tempDirPath = negotiator.drillConfig().getString(ExecConstants.DRILL_TMP_DIR);
    HttpUrl url = buildUrl();
    logger.debug("Final URL: {}", url);
    CustomErrorContext errorContext = new ChildErrorContext(negotiator.parentErrorContext()) {

        @Override
        public void addContext(UserException.Builder builder) {
            super.addContext(builder);
            builder.addContext("URL", url.toString());
        }
    };
    negotiator.setErrorContext(errorContext);
    // Http client setup
    SimpleHttp http = SimpleHttp.builder().scanDefn(subScan).url(url).tempDir(new File(tempDirPath)).proxyConfig(proxySettings(negotiator.drillConfig(), url)).errorContext(errorContext).build();
    // JSON loader setup
    resultSetLoader = negotiator.build();
    if (implicitColumnsAreProjected()) {
        implicitColumns = new ImplicitColumns(resultSetLoader.writer());
        buildImplicitColumns();
    }
    InputStream inStream = http.getInputStream();
    populateImplicitFieldMap(http);
    try {
        JsonLoaderBuilder jsonBuilder = new JsonLoaderBuilder().implicitFields(implicitColumns).resultSetLoader(resultSetLoader).standardOptions(negotiator.queryOptions()).maxRows(maxRecords).dataPath(subScan.tableSpec().connectionConfig().dataPath()).errorContext(errorContext).fromStream(inStream);
        if (subScan.tableSpec().connectionConfig().jsonOptions() != null) {
            JsonLoaderOptions jsonOptions = subScan.tableSpec().connectionConfig().jsonOptions().getJsonOptions(negotiator.queryOptions());
            jsonBuilder.options(jsonOptions);
        } else {
            jsonBuilder.standardOptions(negotiator.queryOptions());
        }
        jsonLoader = jsonBuilder.build();
    } catch (Throwable t) {
        // Paranoia: ensure stream is closed if anything goes wrong.
        // After this, the JSON loader will close the stream.
        AutoCloseables.closeSilently(inStream);
        throw t;
    }
    return true;
}
Also used : SimpleHttp(org.apache.drill.exec.store.http.util.SimpleHttp) InputStream(java.io.InputStream) ChildErrorContext(org.apache.drill.common.exceptions.ChildErrorContext) Builder(okhttp3.HttpUrl.Builder) JsonLoaderBuilder(org.apache.drill.exec.store.easy.json.loader.JsonLoaderImpl.JsonLoaderBuilder) ProxyBuilder(org.apache.drill.exec.store.http.util.HttpProxyConfig.ProxyBuilder) JsonLoaderBuilder(org.apache.drill.exec.store.easy.json.loader.JsonLoaderImpl.JsonLoaderBuilder) JsonLoaderOptions(org.apache.drill.exec.store.easy.json.loader.JsonLoaderOptions) File(java.io.File) HttpUrl(okhttp3.HttpUrl) ImplicitColumns(org.apache.drill.exec.store.ImplicitColumnUtils.ImplicitColumns) CustomErrorContext(org.apache.drill.common.exceptions.CustomErrorContext)

Example 3 with ImplicitColumns

use of org.apache.drill.exec.store.ImplicitColumnUtils.ImplicitColumns in project drill by apache.

the class HttpCSVBatchReader method open.

@Override
public boolean open(SchemaNegotiator negotiator) {
    // Result set loader setup
    String tempDirPath = negotiator.drillConfig().getString(ExecConstants.DRILL_TMP_DIR);
    HttpUrl url = buildUrl();
    CustomErrorContext errorContext = new ChildErrorContext(negotiator.parentErrorContext()) {

        @Override
        public void addContext(UserException.Builder builder) {
            super.addContext(builder);
            builder.addContext("URL", url.toString());
        }
    };
    negotiator.setErrorContext(errorContext);
    // Http client setup
    SimpleHttp http = SimpleHttp.builder().scanDefn(subScan).url(url).tempDir(new File(tempDirPath)).proxyConfig(proxySettings(negotiator.drillConfig(), url)).errorContext(errorContext).build();
    // CSV loader setup
    inStream = http.getInputStream();
    this.csvReader = new CsvParser(csvSettings);
    csvReader.beginParsing(inStream);
    // Build the Schema
    builder = new SchemaBuilder();
    TupleMetadata drillSchema = buildSchema();
    negotiator.tableSchema(drillSchema, true);
    resultLoader = negotiator.build();
    // Add implicit columns
    if (implicitColumnsAreProjected()) {
        implicitColumns = new ImplicitColumns(resultLoader.writer());
        buildImplicitColumns();
        populateImplicitFieldMap(http);
    }
    // Create ScalarWriters
    rowWriter = resultLoader.writer();
    populateWriterArray();
    return true;
}
Also used : SimpleHttp(org.apache.drill.exec.store.http.util.SimpleHttp) ChildErrorContext(org.apache.drill.common.exceptions.ChildErrorContext) SchemaBuilder(org.apache.drill.exec.record.metadata.SchemaBuilder) TupleMetadata(org.apache.drill.exec.record.metadata.TupleMetadata) SchemaBuilder(org.apache.drill.exec.record.metadata.SchemaBuilder) CsvParser(com.univocity.parsers.csv.CsvParser) File(java.io.File) HttpUrl(okhttp3.HttpUrl) ImplicitColumns(org.apache.drill.exec.store.ImplicitColumnUtils.ImplicitColumns) CustomErrorContext(org.apache.drill.common.exceptions.CustomErrorContext)

Aggregations

File (java.io.File)3 HttpUrl (okhttp3.HttpUrl)3 ChildErrorContext (org.apache.drill.common.exceptions.ChildErrorContext)3 CustomErrorContext (org.apache.drill.common.exceptions.CustomErrorContext)3 ImplicitColumns (org.apache.drill.exec.store.ImplicitColumnUtils.ImplicitColumns)3 SimpleHttp (org.apache.drill.exec.store.http.util.SimpleHttp)3 CsvParser (com.univocity.parsers.csv.CsvParser)1 InputStream (java.io.InputStream)1 XMLStreamException (javax.xml.stream.XMLStreamException)1 Builder (okhttp3.HttpUrl.Builder)1 RowSetLoader (org.apache.drill.exec.physical.resultSet.RowSetLoader)1 SchemaBuilder (org.apache.drill.exec.record.metadata.SchemaBuilder)1 TupleMetadata (org.apache.drill.exec.record.metadata.TupleMetadata)1 JsonLoaderBuilder (org.apache.drill.exec.store.easy.json.loader.JsonLoaderImpl.JsonLoaderBuilder)1 JsonLoaderOptions (org.apache.drill.exec.store.easy.json.loader.JsonLoaderOptions)1 ProxyBuilder (org.apache.drill.exec.store.http.util.HttpProxyConfig.ProxyBuilder)1 XMLReader (org.apache.drill.exec.store.xml.XMLReader)1