Search in sources :

Example 1 with StubbingDlpClientFactory

use of com.google.cloud.solutions.autotokenize.testing.stubs.dlp.StubbingDlpClientFactory in project auto-data-tokenize by GoogleCloudPlatform.

the class EncryptionPipelineTest method makeDlpStub.

public void makeDlpStub() {
    if (pipelineOptions.getDlpEncryptConfigJson() == null) {
        return;
    }
    var dlpEncryptConfig = TestResourceLoader.classPath().forProto(DlpEncryptConfig.class).loadJson(configParameters.get("dlpEncryptConfigFile"));
    var encryptSchemaColumns = DeidentifyColumns.columnNamesIn(dlpEncryptConfig);
    expectedSchema = makeExpectedSchema(inputSchema.toString(), encryptSchemaColumns);
    var encryptColumns = encryptSchemaColumns.stream().map(name -> Pattern.compile("^(\\$\\.)?([^\\.]+\\.)(.*)$").matcher(name).replaceAll("$1$3")).collect(toImmutableList());
    dlpClientFactory = new StubbingDlpClientFactory(new Base64EncodingDlpStub(PartialBatchAccumulator.RECORD_ID_COLUMN_NAME, encryptColumns, PROJECT_ID, pipelineOptions.getDlpRegion()));
}
Also used : KeyManagementServiceClient(com.google.cloud.kms.v1.KeyManagementServiceClient) AvroIO(org.apache.beam.sdk.io.AvroIO) FixedClearTextKeySetExtractor(com.google.cloud.solutions.autotokenize.encryptors.FixedClearTextKeySetExtractor) RecordsCountMatcher(com.google.cloud.solutions.autotokenize.testing.RecordsCountMatcher) StringUtils(org.apache.commons.lang3.StringUtils) DlpEncryptConfig(com.google.cloud.solutions.autotokenize.AutoTokenizeMessages.DlpEncryptConfig) SecretsClient(com.google.cloud.solutions.autotokenize.common.SecretsClient) StubbingDlpClientFactory(com.google.cloud.solutions.autotokenize.testing.stubs.dlp.StubbingDlpClientFactory) Collectors.toMap(java.util.stream.Collectors.toMap) PartialBatchAccumulator(com.google.cloud.solutions.autotokenize.dlp.PartialBatchAccumulator) Map(java.util.Map) After(org.junit.After) DeIdentifiedRecordSchemaConverter(com.google.cloud.solutions.autotokenize.common.DeIdentifiedRecordSchemaConverter) Splitter(com.google.common.base.Splitter) Parameterized(org.junit.runners.Parameterized) Schema(org.apache.avro.Schema) ImmutableMap(com.google.common.collect.ImmutableMap) TestDbContainerFactory.makeTestMySQLContainer(com.google.cloud.solutions.autotokenize.testing.TestDbContainerFactory.makeTestMySQLContainer) ImmutableList.toImmutableList(com.google.common.collect.ImmutableList.toImmutableList) Set(java.util.Set) SourceType(com.google.cloud.solutions.autotokenize.AutoTokenizeMessages.SourceType) Collectors(java.util.stream.Collectors) Sets(com.google.common.collect.Sets) Serializable(java.io.Serializable) List(java.util.List) Stream(java.util.stream.Stream) DeidentifyColumns(com.google.cloud.solutions.autotokenize.common.DeidentifyColumns) ConstantSecretVersionValueManagerServicesStub(com.google.cloud.solutions.autotokenize.testing.stubs.secretmanager.ConstantSecretVersionValueManagerServicesStub) Entry(java.util.Map.Entry) Pattern(java.util.regex.Pattern) Joiner(com.google.common.base.Joiner) TestCsvFileGenerator(com.google.cloud.solutions.autotokenize.testing.TestCsvFileGenerator) CsvRowFlatRecordConvertors.makeCsvAvroSchema(com.google.cloud.solutions.autotokenize.common.CsvRowFlatRecordConvertors.makeCsvAvroSchema) RunWith(org.junit.runner.RunWith) Parameters(org.junit.runners.Parameterized.Parameters) Base64DecodingKmsStub(com.google.cloud.solutions.autotokenize.testing.stubs.kms.Base64DecodingKmsStub) PipelineOptionsFactory(org.apache.beam.sdk.options.PipelineOptionsFactory) Function(java.util.function.Function) JdbcDatabaseContainer(org.testcontainers.containers.JdbcDatabaseContainer) Truth(com.google.common.truth.Truth) DlpClientFactory(com.google.cloud.solutions.autotokenize.dlp.DlpClientFactory) ImmutableList(com.google.common.collect.ImmutableList) TestPipeline(org.apache.beam.sdk.testing.TestPipeline) Before(org.junit.Before) TestResourceLoader(com.google.cloud.solutions.autotokenize.testing.TestResourceLoader) GenericRecord(org.apache.avro.generic.GenericRecord) PAssert(org.apache.beam.sdk.testing.PAssert) Base64EncodingDlpStub(com.google.cloud.solutions.autotokenize.testing.stubs.dlp.Base64EncodingDlpStub) Test(org.junit.Test) PCollection(org.apache.beam.sdk.values.PCollection) Collectors.toList(java.util.stream.Collectors.toList) Rule(org.junit.Rule) TemporaryFolder(org.junit.rules.TemporaryFolder) StubbingDlpClientFactory(com.google.cloud.solutions.autotokenize.testing.stubs.dlp.StubbingDlpClientFactory) Base64EncodingDlpStub(com.google.cloud.solutions.autotokenize.testing.stubs.dlp.Base64EncodingDlpStub) DlpEncryptConfig(com.google.cloud.solutions.autotokenize.AutoTokenizeMessages.DlpEncryptConfig)

Example 2 with StubbingDlpClientFactory

use of com.google.cloud.solutions.autotokenize.testing.stubs.dlp.StubbingDlpClientFactory in project auto-data-tokenize by GoogleCloudPlatform.

the class BatchAndDlpDeIdRecordsTest method expand_valid.

@Test
public void expand_valid() {
    ImmutableList<FlatRecord> expectedBase64EncodedContacts = TestResourceLoader.classPath().forProto(FlatRecord.class).loadAllTextFiles(ImmutableList.of("avro_records/contacts_schema/jane_doe_contact_number_base64_avro_record.textpb", "avro_records/contacts_schema/john_doe_contact_number_base64_avro_record.textpb"));
    PCollection<FlatRecord> tokenizedRecords = testPipeline.apply(Create.of(CONTACT_RECORDS)).apply(BatchAndDlpDeIdRecords.withEncryptConfig(NUMBER_TOKENIZE_CONFIG).withDlpProjectId("dlp-test-project").withDlpRegion(testDlpRegion).withDlpClientFactory(new StubbingDlpClientFactory(new Base64EncodingDlpStub(PartialBatchAccumulator.RECORD_ID_COLUMN_NAME, ImmutableList.of("$.contacts.[\"contact\"].number"), "dlp-test-project", testDlpRegion))));
    PAssert.that(tokenizedRecords).satisfies(new FlatRecordCheckerWithoutRecordIds(expectedBase64EncodedContacts));
    testPipeline.run().waitUntilFinish();
}
Also used : StubbingDlpClientFactory(com.google.cloud.solutions.autotokenize.testing.stubs.dlp.StubbingDlpClientFactory) Base64EncodingDlpStub(com.google.cloud.solutions.autotokenize.testing.stubs.dlp.Base64EncodingDlpStub) FlatRecord(com.google.cloud.solutions.autotokenize.AutoTokenizeMessages.FlatRecord) Test(org.junit.Test)

Aggregations

Base64EncodingDlpStub (com.google.cloud.solutions.autotokenize.testing.stubs.dlp.Base64EncodingDlpStub)2 StubbingDlpClientFactory (com.google.cloud.solutions.autotokenize.testing.stubs.dlp.StubbingDlpClientFactory)2 KeyManagementServiceClient (com.google.cloud.kms.v1.KeyManagementServiceClient)1 DlpEncryptConfig (com.google.cloud.solutions.autotokenize.AutoTokenizeMessages.DlpEncryptConfig)1 FlatRecord (com.google.cloud.solutions.autotokenize.AutoTokenizeMessages.FlatRecord)1 SourceType (com.google.cloud.solutions.autotokenize.AutoTokenizeMessages.SourceType)1 CsvRowFlatRecordConvertors.makeCsvAvroSchema (com.google.cloud.solutions.autotokenize.common.CsvRowFlatRecordConvertors.makeCsvAvroSchema)1 DeIdentifiedRecordSchemaConverter (com.google.cloud.solutions.autotokenize.common.DeIdentifiedRecordSchemaConverter)1 DeidentifyColumns (com.google.cloud.solutions.autotokenize.common.DeidentifyColumns)1 SecretsClient (com.google.cloud.solutions.autotokenize.common.SecretsClient)1 DlpClientFactory (com.google.cloud.solutions.autotokenize.dlp.DlpClientFactory)1 PartialBatchAccumulator (com.google.cloud.solutions.autotokenize.dlp.PartialBatchAccumulator)1 FixedClearTextKeySetExtractor (com.google.cloud.solutions.autotokenize.encryptors.FixedClearTextKeySetExtractor)1 RecordsCountMatcher (com.google.cloud.solutions.autotokenize.testing.RecordsCountMatcher)1 TestCsvFileGenerator (com.google.cloud.solutions.autotokenize.testing.TestCsvFileGenerator)1 TestDbContainerFactory.makeTestMySQLContainer (com.google.cloud.solutions.autotokenize.testing.TestDbContainerFactory.makeTestMySQLContainer)1 TestResourceLoader (com.google.cloud.solutions.autotokenize.testing.TestResourceLoader)1 Base64DecodingKmsStub (com.google.cloud.solutions.autotokenize.testing.stubs.kms.Base64DecodingKmsStub)1 ConstantSecretVersionValueManagerServicesStub (com.google.cloud.solutions.autotokenize.testing.stubs.secretmanager.ConstantSecretVersionValueManagerServicesStub)1 Joiner (com.google.common.base.Joiner)1