Search in sources :

Example 1 with OAIHarvestConfiguration

use of cz.mzk.recordmanager.server.model.OAIHarvestConfiguration in project RecordManager2 by moravianlibrary.

the class AsyncOAIItemReader method beforeStep.

@Override
public void beforeStep(final StepExecution stepExecution) {
    try (SessionBinder sess = hibernateSync.register()) {
        OAIHarvestConfiguration conf = configDao.get(confId);
        OAIHarvesterParams params = new OAIHarvesterParams();
        params.setUrl(conf.getUrl());
        params.setMetadataPrefix(conf.getMetadataPrefix());
        params.setGranularity(conf.getGranularity());
        params.setSet(conf.getSet());
        params.setFrom(fromDate);
        params.setUntil(untilDate);
        harvester = harvesterFactory.create(params);
        processIdentify(conf);
        conf = configDao.get(confId);
        params.setGranularity(conf.getGranularity());
        harvester = harvesterFactory.create(params);
    }
}
Also used : OAIHarvestConfiguration(cz.mzk.recordmanager.server.model.OAIHarvestConfiguration) SessionBinder(cz.mzk.recordmanager.server.util.HibernateSessionSynchronizer.SessionBinder)

Example 2 with OAIHarvestConfiguration

use of cz.mzk.recordmanager.server.model.OAIHarvestConfiguration in project RecordManager2 by moravianlibrary.

the class HarvestingFacadeTest method harvest.

@Test
public void harvest() throws IOException {
    reset(httpClient);
    final Capture<String> url = EasyMock.newCapture();
    expect(httpClient.executeGet(capture(url))).andAnswer(() -> reply(url.getValue())).anyTimes();
    replay(httpClient);
    OAIHarvestConfiguration conf = oaiHarvestConfigurationDao.get(300L);
    harvestingFacade.incrementalHarvest(conf);
    harvestingFacade.incrementalHarvest(conf);
}
Also used : OAIHarvestConfiguration(cz.mzk.recordmanager.server.model.OAIHarvestConfiguration) Test(org.testng.annotations.Test) AbstractTest(cz.mzk.recordmanager.server.AbstractTest)

Example 3 with OAIHarvestConfiguration

use of cz.mzk.recordmanager.server.model.OAIHarvestConfiguration in project RecordManager2 by moravianlibrary.

the class CosmotronHarvestJobTest method test996BeforeRecord.

@Test
public void test996BeforeRecord() throws Exception {
    reset(httpClient);
    InputStream response0 = this.getClass().getResourceAsStream("/sample/Identify.xml");
    InputStream response1 = this.getClass().getResourceAsStream("/sample/cosmotron/996BeforeRecord.xml");
    expect(httpClient.executeGet("http://katalog.cbvk.cz/i2/i2.ws.oai.cls?verb=Identify")).andReturn(response0);
    expect(httpClient.executeGet("http://katalog.cbvk.cz/i2/i2.ws.oai.cls?verb=ListRecords&metadataPrefix=oai_marcxml_cpk")).andReturn(response1);
    replay(httpClient);
    final Long confID = 328L;
    Map<String, JobParameter> params = new HashMap<>();
    params.put(Constants.JOB_PARAM_CONF_ID, new JobParameter(confID));
    JobExecution exec = jobExecutor.execute(Constants.JOB_ID_HARVEST_COSMOTRON, new JobParameters(params));
    Assert.assertEquals(exec.getExitStatus(), ExitStatus.COMPLETED);
    OAIHarvestConfiguration config = configDao.get(confID);
    HarvestedRecord hr = recordDao.findByIdAndHarvestConfiguration("CbvkUsCat" + Constants.COSMOTRON_RECORD_ID_CHAR + "m0000002", config);
    Assert.assertNotNull(hr);
    Assert.assertNotNull(cosmotronDao.findByIdAndHarvestConfiguration("CbvkUsCat" + Constants.COSMOTRON_RECORD_ID_CHAR + "0000003", config));
    Assert.assertNotNull(cosmotronDao.findByIdAndHarvestConfiguration("CbvkUsCat" + Constants.COSMOTRON_RECORD_ID_CHAR + "0000004", config));
    Assert.assertNotNull(cosmotronDao.findByIdAndHarvestConfiguration("CbvkUsCat" + Constants.COSMOTRON_RECORD_ID_CHAR + "0000005", config));
    InputStream is = new ByteArrayInputStream(hr.getRawRecord());
    Record record = marcXmlParser.parseUnderlyingRecord(is);
    MarcRecord marcRecord = new MarcRecordImpl(record);
    Assert.assertEquals(marcRecord.getDataFields("996").size(), 3);
}
Also used : OAIHarvestConfiguration(cz.mzk.recordmanager.server.model.OAIHarvestConfiguration) HashMap(java.util.HashMap) ByteArrayInputStream(java.io.ByteArrayInputStream) InputStream(java.io.InputStream) MarcRecord(cz.mzk.recordmanager.server.marc.MarcRecord) JobExecution(org.springframework.batch.core.JobExecution) MarcRecordImpl(cz.mzk.recordmanager.server.marc.MarcRecordImpl) ByteArrayInputStream(java.io.ByteArrayInputStream) MarcRecord(cz.mzk.recordmanager.server.marc.MarcRecord) Record(org.marc4j.marc.Record) HarvestedRecord(cz.mzk.recordmanager.server.model.HarvestedRecord) JobParameters(org.springframework.batch.core.JobParameters) JobParameter(org.springframework.batch.core.JobParameter) HarvestedRecord(cz.mzk.recordmanager.server.model.HarvestedRecord) Test(org.testng.annotations.Test) AbstractTest(cz.mzk.recordmanager.server.AbstractTest)

Example 4 with OAIHarvestConfiguration

use of cz.mzk.recordmanager.server.model.OAIHarvestConfiguration in project RecordManager2 by moravianlibrary.

the class CosmotronHarvestJobTest method testRecordWithParentIdAndWithout996.

@Test
public void testRecordWithParentIdAndWithout996() throws Exception {
    reset(httpClient);
    InputStream response0 = this.getClass().getResourceAsStream("/sample/Identify.xml");
    InputStream response1 = this.getClass().getResourceAsStream("/sample/cosmotron/ParentIdNot996.xml");
    expect(httpClient.executeGet("http://katalog.cbvk.cz/i2/i2.ws.oai.cls?verb=Identify")).andReturn(response0);
    expect(httpClient.executeGet("http://katalog.cbvk.cz/i2/i2.ws.oai.cls?verb=ListRecords&metadataPrefix=oai_marcxml_cpk")).andReturn(response1);
    replay(httpClient);
    final Long confID = 328L;
    Map<String, JobParameter> params = new HashMap<>();
    params.put(Constants.JOB_PARAM_CONF_ID, new JobParameter(confID));
    JobExecution exec = jobExecutor.execute(Constants.JOB_ID_HARVEST_COSMOTRON, new JobParameters(params));
    Assert.assertEquals(exec.getExitStatus(), ExitStatus.COMPLETED);
    OAIHarvestConfiguration config = configDao.get(confID);
    Cosmotron996 cosmoRec = cosmotronDao.findByIdAndHarvestConfiguration("CbvkUsCat_0000003", config);
    Assert.assertNotNull(cosmoRec);
    Assert.assertNotNull(cosmoRec.getDeleted());
}
Also used : JobExecution(org.springframework.batch.core.JobExecution) OAIHarvestConfiguration(cz.mzk.recordmanager.server.model.OAIHarvestConfiguration) HashMap(java.util.HashMap) Cosmotron996(cz.mzk.recordmanager.server.model.Cosmotron996) ByteArrayInputStream(java.io.ByteArrayInputStream) InputStream(java.io.InputStream) JobParameters(org.springframework.batch.core.JobParameters) JobParameter(org.springframework.batch.core.JobParameter) Test(org.testng.annotations.Test) AbstractTest(cz.mzk.recordmanager.server.AbstractTest)

Example 5 with OAIHarvestConfiguration

use of cz.mzk.recordmanager.server.model.OAIHarvestConfiguration in project RecordManager2 by moravianlibrary.

the class CosmotronHarvestJobTest method testNew996.

@Test
public void testNew996() throws Exception {
    reset(httpClient);
    InputStream response0 = this.getClass().getResourceAsStream("/sample/Identify.xml");
    InputStream response1 = this.getClass().getResourceAsStream("/sample/cosmotron/New9961.xml");
    InputStream response2 = this.getClass().getResourceAsStream("/sample/cosmotron/New9962.xml");
    expect(httpClient.executeGet("http://katalog.cbvk.cz/i2/i2.ws.oai.cls?verb=Identify")).andReturn(response0);
    expect(httpClient.executeGet("http://katalog.cbvk.cz/i2/i2.ws.oai.cls?verb=ListRecords&metadataPrefix=oai_marcxml_cpk")).andReturn(response1);
    expect(httpClient.executeGet("http://katalog.cbvk.cz/i2/i2.ws.oai.cls?verb=ListRecords&resumptionToken=12345")).andReturn(response2);
    replay(httpClient);
    final Long confID = 328L;
    Map<String, JobParameter> params = new HashMap<>();
    params.put(Constants.JOB_PARAM_CONF_ID, new JobParameter(confID));
    JobExecution exec = jobExecutor.execute(Constants.JOB_ID_HARVEST_COSMOTRON, new JobParameters(params));
    Assert.assertEquals(exec.getExitStatus(), ExitStatus.COMPLETED);
    OAIHarvestConfiguration config = configDao.get(confID);
    HarvestedRecord hr = recordDao.findByIdAndHarvestConfiguration("CbvkUsCat" + Constants.COSMOTRON_RECORD_ID_CHAR + "m0000002", config);
    Assert.assertNotNull(hr);
    Assert.assertNotNull(cosmotronDao.findByIdAndHarvestConfiguration("CbvkUsCat" + Constants.COSMOTRON_RECORD_ID_CHAR + "0000003", config));
    Assert.assertNotNull(cosmotronDao.findByIdAndHarvestConfiguration("CbvkUsCat" + Constants.COSMOTRON_RECORD_ID_CHAR + "0000004", config));
    InputStream is = new ByteArrayInputStream(hr.getRawRecord());
    Record record = marcXmlParser.parseUnderlyingRecord(is);
    MarcRecord marcRecord = new MarcRecordImpl(record);
    Assert.assertEquals(marcRecord.getDataFields("996").size(), 5);
}
Also used : OAIHarvestConfiguration(cz.mzk.recordmanager.server.model.OAIHarvestConfiguration) HashMap(java.util.HashMap) ByteArrayInputStream(java.io.ByteArrayInputStream) InputStream(java.io.InputStream) MarcRecord(cz.mzk.recordmanager.server.marc.MarcRecord) JobExecution(org.springframework.batch.core.JobExecution) MarcRecordImpl(cz.mzk.recordmanager.server.marc.MarcRecordImpl) ByteArrayInputStream(java.io.ByteArrayInputStream) MarcRecord(cz.mzk.recordmanager.server.marc.MarcRecord) Record(org.marc4j.marc.Record) HarvestedRecord(cz.mzk.recordmanager.server.model.HarvestedRecord) JobParameters(org.springframework.batch.core.JobParameters) JobParameter(org.springframework.batch.core.JobParameter) HarvestedRecord(cz.mzk.recordmanager.server.model.HarvestedRecord) Test(org.testng.annotations.Test) AbstractTest(cz.mzk.recordmanager.server.AbstractTest)

Aggregations

OAIHarvestConfiguration (cz.mzk.recordmanager.server.model.OAIHarvestConfiguration)18 AbstractTest (cz.mzk.recordmanager.server.AbstractTest)12 Test (org.testng.annotations.Test)12 InputStream (java.io.InputStream)9 HashMap (java.util.HashMap)9 JobExecution (org.springframework.batch.core.JobExecution)9 JobParameter (org.springframework.batch.core.JobParameter)9 JobParameters (org.springframework.batch.core.JobParameters)9 ByteArrayInputStream (java.io.ByteArrayInputStream)7 HarvestedRecord (cz.mzk.recordmanager.server.model.HarvestedRecord)6 SessionBinder (cz.mzk.recordmanager.server.util.HibernateSessionSynchronizer.SessionBinder)6 MarcRecord (cz.mzk.recordmanager.server.marc.MarcRecord)4 MarcRecordImpl (cz.mzk.recordmanager.server.marc.MarcRecordImpl)4 Record (org.marc4j.marc.Record)4 Cosmotron996 (cz.mzk.recordmanager.server.model.Cosmotron996)3 DownloadImportConfiguration (cz.mzk.recordmanager.server.model.DownloadImportConfiguration)2 RegexpExtractor (cz.mzk.recordmanager.server.util.RegexpExtractor)2 ImportConfiguration (cz.mzk.recordmanager.server.model.ImportConfiguration)1 KrameriusConfiguration (cz.mzk.recordmanager.server.model.KrameriusConfiguration)1 TransformerConfigurationException (javax.xml.transform.TransformerConfigurationException)1