Search in sources :

Example 1 with JdbcDatasource

use of com.thinkbiganalytics.spark.rest.model.JdbcDatasource in project kylo by Teradata.

the class AbstractDatasourceProvider method getTableFromDatasource.

@Nonnull
@Override
public final T getTableFromDatasource(@Nonnull final String table, @Nonnull final Datasource datasource, @Nonnull final SQLContext sqlContext) {
    if (datasource instanceof JdbcDatasource) {
        final JdbcDatasource jdbcDatasource = (JdbcDatasource) datasource;
        final Properties properties = new Properties();
        properties.put("driver", jdbcDatasource.getDatabaseDriverClassName());
        if (StringUtils.isNotBlank(jdbcDatasource.getDatabaseUser())) {
            properties.put("user", jdbcDatasource.getDatabaseUser());
        }
        if (StringUtils.isNotBlank(jdbcDatasource.getPassword())) {
            properties.put("password", jdbcDatasource.getPassword());
        }
        return readJdbcTable(jdbcDatasource.getDatabaseConnectionUrl(), table, properties, sqlContext);
    } else {
        throw new IllegalArgumentException("Datasource does not provide tables: " + datasource);
    }
}
Also used : JdbcDatasource(com.thinkbiganalytics.spark.rest.model.JdbcDatasource) Properties(java.util.Properties) Nonnull(javax.annotation.Nonnull)

Example 2 with JdbcDatasource

use of com.thinkbiganalytics.spark.rest.model.JdbcDatasource in project kylo by Teradata.

the class TransformService method saveSql.

/**
 * Executes and saves a Spark SQL request.
 */
@Nonnull
public SaveResponse saveSql(@Nonnull final String id, @Nonnull final SaveRequest save) {
    log.entry(id, save);
    // Create task
    final Supplier<SaveResult> task;
    final TransformRequest transform = getTransformRequest(id);
    final JdbcDatasource transformDatasource = (transform.getDatasources() != null && transform.getDatasources().size() == 1 && transform.getDatasources().get(0) instanceof JdbcDatasource) ? (JdbcDatasource) transform.getDatasources().get(0) : null;
    if (transformDatasource != null && save.getJdbc() != null && Objects.equal(transformDatasource.getId(), save.getJdbc().getId())) {
        Preconditions.checkArgument(save.getTableName() != null, "Missing target table name.");
        task = new SaveSqlStage(save.getTableName(), transform.getScript(), save.getJdbc());
    } else {
        task = createSaveTask(save, createSqlTask(transform));
    }
    // Submit job
    final SaveResponse response = submitSaveJob(task);
    return log.exit(response);
}
Also used : JdbcDatasource(com.thinkbiganalytics.spark.rest.model.JdbcDatasource) SaveSqlStage(com.thinkbiganalytics.spark.metadata.SaveSqlStage) SaveResponse(com.thinkbiganalytics.spark.rest.model.SaveResponse) SaveResult(com.thinkbiganalytics.spark.model.SaveResult) TransformRequest(com.thinkbiganalytics.spark.rest.model.TransformRequest) Nonnull(javax.annotation.Nonnull)

Example 3 with JdbcDatasource

use of com.thinkbiganalytics.spark.rest.model.JdbcDatasource in project kylo by Teradata.

the class SparkShellProxyController method resolveDatasources.

/**
 * Retrieves all details of the specified data sources.
 */
@Nonnull
private List<Datasource> resolveDatasources(@Nonnull final List<Datasource> sources) {
    // Verify access to data sources
    accessController.checkPermission(AccessController.SERVICES, FeedServicesAccessControl.ACCESS_DATASOURCES);
    final List<com.thinkbiganalytics.metadata.api.datasource.Datasource.ID> datasourceIds = metadata.read(() -> sources.stream().map(com.thinkbiganalytics.metadata.datasource.Datasource::getId).map(datasourceProvider::resolve).map(id -> {
        final com.thinkbiganalytics.metadata.api.datasource.Datasource datasource = datasourceProvider.getDatasource(id);
        if (datasource != null) {
            return datasource.getId();
        } else {
            throw new BadRequestException("No datasource exists with the given ID: " + id);
        }
    }).collect(Collectors.toList()));
    // Retrieve admin-level details
    return metadata.read(() -> datasourceIds.stream().map(datasourceProvider::getDatasource).map(datasource -> {
        if (datasource instanceof com.thinkbiganalytics.metadata.api.datasource.UserDatasource) {
            return (com.thinkbiganalytics.metadata.datasource.Datasource) datasourceTransform.toDatasource(datasource, DatasourceModelTransform.Level.ADMIN);
        } else {
            throw new BadRequestException("Not a supported datasource: " + datasource.getClass().getSimpleName() + " " + datasource.getId());
        }
    }).map(datasource -> {
        if (datasource instanceof com.thinkbiganalytics.metadata.datasource.JdbcDatasource) {
            return new JdbcDatasource((com.thinkbiganalytics.metadata.datasource.JdbcDatasource) datasource);
        } else {
            throw new BadRequestException("Not a supported datasource: " + datasource.getClass().getSimpleName());
        }
    }).collect(Collectors.toList()), MetadataAccess.SERVICE);
}
Also used : Datasource(com.thinkbiganalytics.spark.rest.model.Datasource) JdbcDatasource(com.thinkbiganalytics.spark.rest.model.JdbcDatasource) TransformRequest(com.thinkbiganalytics.spark.rest.model.TransformRequest) Produces(javax.ws.rs.Produces) LoggerFactory(org.slf4j.LoggerFactory) Path(javax.ws.rs.Path) DatasourceModelTransform(com.thinkbiganalytics.feedmgr.service.datasource.DatasourceModelTransform) User(org.springframework.security.core.userdetails.User) ApiParam(io.swagger.annotations.ApiParam) ApiOperation(io.swagger.annotations.ApiOperation) MediaType(javax.ws.rs.core.MediaType) SaveRequest(com.thinkbiganalytics.spark.rest.model.SaveRequest) Consumes(javax.ws.rs.Consumes) AccessController(com.thinkbiganalytics.security.AccessController) BadRequestException(javax.ws.rs.BadRequestException) FeedServicesAccessControl(com.thinkbiganalytics.feedmgr.security.FeedServicesAccessControl) MetadataAccess(com.thinkbiganalytics.metadata.api.MetadataAccess) SecurityContextHolder(org.springframework.security.core.context.SecurityContextHolder) SparkShellRestClient(com.thinkbiganalytics.spark.shell.SparkShellRestClient) MissingResourceException(java.util.MissingResourceException) Collectors(java.util.stream.Collectors) List(java.util.List) HttpHeaders(javax.ws.rs.core.HttpHeaders) Response(javax.ws.rs.core.Response) SaveResponse(com.thinkbiganalytics.spark.rest.model.SaveResponse) Optional(java.util.Optional) WebApplicationException(javax.ws.rs.WebApplicationException) Pattern(java.util.regex.Pattern) Authentication(org.springframework.security.core.Authentication) PathParam(javax.ws.rs.PathParam) GET(javax.ws.rs.GET) ApiResponses(io.swagger.annotations.ApiResponses) Supplier(java.util.function.Supplier) Inject(javax.inject.Inject) TransformResponse(com.thinkbiganalytics.spark.rest.model.TransformResponse) RestResponseStatus(com.thinkbiganalytics.rest.model.RestResponseStatus) ResourceBundle(java.util.ResourceBundle) Datasource(com.thinkbiganalytics.spark.rest.model.Datasource) Api(io.swagger.annotations.Api) Tag(io.swagger.annotations.Tag) Nonnull(javax.annotation.Nonnull) Nullable(javax.annotation.Nullable) Logger(org.slf4j.Logger) POST(javax.ws.rs.POST) SparkShellSaveException(com.thinkbiganalytics.spark.shell.SparkShellSaveException) SparkShellProcess(com.thinkbiganalytics.spark.shell.SparkShellProcess) JdbcDatasource(com.thinkbiganalytics.spark.rest.model.JdbcDatasource) SparkShellProcessManager(com.thinkbiganalytics.spark.shell.SparkShellProcessManager) Component(org.springframework.stereotype.Component) SwaggerDefinition(io.swagger.annotations.SwaggerDefinition) ApiResponse(io.swagger.annotations.ApiResponse) RegistrationRequest(com.thinkbiganalytics.spark.rest.model.RegistrationRequest) DatasourceProvider(com.thinkbiganalytics.metadata.api.datasource.DatasourceProvider) SparkShellTransformException(com.thinkbiganalytics.spark.shell.SparkShellTransformException) Collections(java.util.Collections) JdbcDatasource(com.thinkbiganalytics.spark.rest.model.JdbcDatasource) BadRequestException(javax.ws.rs.BadRequestException) Nonnull(javax.annotation.Nonnull)

Aggregations

JdbcDatasource (com.thinkbiganalytics.spark.rest.model.JdbcDatasource)3 Nonnull (javax.annotation.Nonnull)3 SaveResponse (com.thinkbiganalytics.spark.rest.model.SaveResponse)2 TransformRequest (com.thinkbiganalytics.spark.rest.model.TransformRequest)2 FeedServicesAccessControl (com.thinkbiganalytics.feedmgr.security.FeedServicesAccessControl)1 DatasourceModelTransform (com.thinkbiganalytics.feedmgr.service.datasource.DatasourceModelTransform)1 MetadataAccess (com.thinkbiganalytics.metadata.api.MetadataAccess)1 DatasourceProvider (com.thinkbiganalytics.metadata.api.datasource.DatasourceProvider)1 RestResponseStatus (com.thinkbiganalytics.rest.model.RestResponseStatus)1 AccessController (com.thinkbiganalytics.security.AccessController)1 SaveSqlStage (com.thinkbiganalytics.spark.metadata.SaveSqlStage)1 SaveResult (com.thinkbiganalytics.spark.model.SaveResult)1 Datasource (com.thinkbiganalytics.spark.rest.model.Datasource)1 RegistrationRequest (com.thinkbiganalytics.spark.rest.model.RegistrationRequest)1 SaveRequest (com.thinkbiganalytics.spark.rest.model.SaveRequest)1 TransformResponse (com.thinkbiganalytics.spark.rest.model.TransformResponse)1 SparkShellProcess (com.thinkbiganalytics.spark.shell.SparkShellProcess)1 SparkShellProcessManager (com.thinkbiganalytics.spark.shell.SparkShellProcessManager)1 SparkShellRestClient (com.thinkbiganalytics.spark.shell.SparkShellRestClient)1 SparkShellSaveException (com.thinkbiganalytics.spark.shell.SparkShellSaveException)1