How to separately generate each ADF copy Pipeline to load single SAT, HUB and PIT, currently one batch clubs multiple copy pipelines.
Our requirement is to load each batch having single copy task so that we can trigger it end to end parallelly till RDV layers, instead of running multiply copy pipelines with single batch.
we have events(JSON) coming to Azure Storage account through multiple different APIs, that gets flattened out by a process and loaded to SQL DB STG (STG has CDC enabled for Biml generated ADF to identify new records), after STG we trigger ADF to load data from STG to RDV (SAT, HUB and PIT), as explained above BIML has clubbed multiple copy pipeline inside single batch , and because of that we have to run all of them together to load data till RDV, but requirement is to load data end to end till RDV for each API independently.
How to achieve this using Bilmflex ?
