Where data in interfaced out of the system, there's always the discussion of full load and the delta loads, with advantages and disadvantages on both flavors. This article describes how to set up shadow checking in a way that I've implemented on 10+ serious interfaces to date.
The setup
Outbound interfaces are best compared to a report - data is selected, formatted in some way and presented. Not in a report but in a file. It's the easy end of interfacing, because the data is simply gathered and output. The real work of an interface is the inbound side: where data needs to be processed. The shadow tracking setup is aimed at the outbound interface.
In most "delta" setups, the data that was changed/renewed/added since the last time the interface ran are selected. This can often be done by selections with creation and change dates, or e.g. checking change documents for the data you are interfacing. The shadow tracking setup adds a bit more control to your interface.
What shadow tracking does is best explained in an example:
- The interface report is started, data is selected and sculptured into the format that is required by the receiver
For this example, an .CSV file is composed with a simple 20-column set of data - A single line of data is composed, which has a key, such as customer number or customer number and company code
- The shadow table is checked with the key
- If there are no differences - the line is ignored
- If there are differences, the line is placed in the interface output file and the shadow image is updated for the key
This way of working captures changes irrelevant of where the change came from. There is no need to rely on creation or change dates the system may or may not have for your data.
All information that is sent out as data on the interface - is regarded worthy of a change trigger. Whatever data changes on the data to be delivered, the shadow mechanism will pick it up and resend it.
The concept - by example
The slide below shows how all involved parties have allowed themselves to be laid out in a single slide. SAP is the outgoing party, and it's delivers customer information for the first time. As there is no shadow table yet, the first data-delivery will not just deliver data, but also fill the shadow table.
This next slide reveals where things could go wrong - just to do an impact assessment:
- Startup failure: for some reason the daily job that should have run yesterday, didn't. This merely means that today's job will also process yesterdays work - and nothing gets lost. Net effect: nothing more than delay
- File missing in transfer - that's really an Achilles heel thing here: missing files will result in relevant data updates. If you were operating with full loads, the missing file matter wouldn't be to bad - as the next delivery will hold the same information again, keeping the systems aligned.
- Then there's the errors are the receiving end - if data can not be processed, there is another chance of change losses.
To tackle these differences, there is a "resend since" option on the shadow approach. A redo of e.g. a week or a month can be done sending all changes since last month through the interface. Total control !
Shadow image data - how ?
The focus for shadow tracking is the object to be shadowed. In a .csv interface this would be a line on the file, which isa set of customer settings for a company code in the example used here. SAP has an EXPORT TO DATABASE setup where data of any sort is exported for a certain key. This key would be something line KUNNR~0000012345~1200 where the customer number and company code form the key. When a full run is done, all customer numbers will create an entry on the shadow table for every company code.
The actual shadow image is in fact an Abap internal structure or even table (internal table) that is exported / stored. The so called INDX tables (there are many of these tables) will hold anything that is handed to it. For the shadow tracking setup it's best to create a new INDX table, as relevant fields like the originating report, the creation date/time and the last changed date/time are important.
Aging shadows
The shadow table will be created on day 1, and will grow a day older every day like you and me. Even though a shadow table entry can itself contain records or even tables of records - if there are no actual changes made, the shadow will remain unchanged. This is where the importance of the create date and the change date comes in: it states when the shadow image was created, but more importantly, it also reflects when it was last changed. This allows us to produce a "partial full load" of the data. Here's how:
Say for example we are sending customer data to a CMS system twice a day, changes only. As the customer data includes master data as well as payment data, every delivery holds 20 to 30 sets of customer data. With 2 files/deliveries per day, we want to ensure everything was processed and no files were lost, by re-sending all data again at the end of the month - as a full load. Simple enough to do, but a full load would also include the customers that have been idle for over 5 years.
Here's the added dimension of maintaining a change-date on the shadow tables: the interface can be selective about which shadow information it wants to involve. So simply run the interface with the "Check" indicator on, the "Update" indicator off and supply the number of days you want to "resend". The interface will then calculate a "most recent" date from the number of days to be resend, then compose it's output and check the shadow information - providing the shadow information is not too new. A redend of all of the changes of last month. Or last week.
There is something important to realize about resends - the "Update" indicator. From sending perspective, a resend gathers data that was already processed before. The reason for resending information was the assumption that not everything that was delivered has also been received. So when a resend is done with the "Update" indicator on, the shadow table entries will be updated with todays information and todays change date and time. This will disrupt the "normal flow". Ergo: use resend = don't set the Update indicator.
Step-by-step implementation
So how is this all implemented ? A step plan:
Step 1. Build your own outbound interface
Simply get to work and build your interface. Or if are done already or if you want to add shadow tracking to an existing interface, skip this step. There is one thing: you will need to choose a key for your shadow image. Have a quick peek at step 7. to get an idea of how shadow and actual data should be processed.
Step 2. Create the shadow table
This step is only needed for the first interface you will enable shadow controls on. A single table can work with all the interfaces you want. Create table ZSHADOW_INDX with the following fields. Note that table name and field names are important, as they should align with the rest of the steps.
| MANDT | MANDT |
| RELID | INDX_RELID |
| SRTFD | INDX_SRTFD |
| SRTF2 | INDX_SRTF2 |
| CREATE_DATE | SYDATS |
| CREATE_TIME | SYTIME |
| CHANGE_DATE | SYDATS |
| CHANGE_TIME | SYTIME |
| PROGNAME | PROGNAME |
| CLUSTR | INDX_CLSTR |
| CLUSTD | INDX_CLUST |
Step 3. Add controls to your selection screen
By simply adding the coding below. This will effectively compose a line with the 2 controls checkboxes "Check" and "Update" which allows you to do all you need.
SELECTION-SCREEN: SKIP, BEGIN OF LINE, COMMENT 1(33) lbl_s1. PARAMETERS: pa_shaco AS CHECKBOX default abap_true. SELECTION-SCREEN: COMMENT 38(14) lbl_s2 FOR FIELD pa_shaco. parameters: pa_shaup AS CHECKBOX default abap_true. SELECTION-SCREEN: COMMENT 56(20) lbl_s3 FOR FIELD pa_shaup, END OF LINE, BEGIN OF LINE, COMMENT 1(33) lbl_s4. parameters: pa_share type d. SELECTION-SCREEN: END OF LINE.
With these settings, some text labels also need to be filled in, on INITIALIZATION:
lbl_s1 = 'Shadow table (change control)'. lbl_s2 = 'Check'. lbl_s3 = 'Update'. lbl_s4 = 'Resend since'.
Then there is a little backdoor functionality, which you will also need to implement. Here's the thing: your super interface will be de-commissioned some day. And with proper software comes de-installation functionality, which is a valid point for your interface. What should happen to the shadow table information, once your interface is no longer used ? Right: it should be cleared.
And there is another very valid reason for this backdoor: before the interface is scrapped, someone will ask you to add a few fields. Maybe even a while file. The structure of the data that is stored on the shadow table may become obsolete, in which case you will get read errors on the import. Another great reason to have the functionality to clear the shadow table.
AT SELECTION-SCREEN.
CASE sscrfields-ucomm.
* Please note: this function code is not made available on the selection screen
* for an end user to find. It is for maintenance purpose only, allowing the
* setup to be taken out of commission or to accomodate test cycles. This is
* "hidden" functionality.
WHEN 'RESET_SHADOW'.
lcl_shadow=>gr_kunnr[] = so_kunnr[].
lcl_shadow=>reset( ).
ENDCASE.
Step 4. Implement the lcl_Shadow class
Below you will find the definition and implementation of the lcl_Shadow class, which takes care of fetching and storing the shadow data. Implement it in your own coding as an include or as part of the main body. If you're not an Object Oriented minded programmer, put it in anyway.
CLASS lcl_shadow DEFINITION FINAL.
PUBLIC SECTION.
types:
begin of ty_masterdata,
kunnr type kna1-kunnr,
content type string,
end of ty_masterdata,
begin of ty_coco,
bukrs type knb1-bukrs,
content type string,
end of ty_coco,
begin of ty_shadow,
kunnr type kna1-kunnr,
masterdata type ty_masterdata,
coco type STANDARD TABLE OF ty_coco with default key,
end of ty_shadow.
CLASS-DATA:
gv_CHECK type boolean,
gv_UPDATE type boolean,
gv_RESEND_DATE type d,
gw_shadow type ty_shadow, "Work area
gw_shadow_indx type zshadow_indx,
gr_kunnr type range of kna1-kunnr.
CLASS-METHODS:
reset,
read importing kunnr type kna1-kunnr,
save.
ENDCLASS.
The implementation of the class, where the actual type definition of the shadow table data is already set. Adjust to your own needs and requirements.
CLASS lcl_shadow IMPLEMENTATION.
method reset.
data: lv_answer type c length 1,
lt_SRTFD type standard table of INDX_SRTFD,
lv_SRTFD type INDX_SRTFD,
begin of lv_SRTFD_fields,
dummy type c length 10,
kunnr type c length 10,
end of lv_SRTFD_fields.
* This method will actively remove entries from the shadow tables, which is
* "backdoor" funtionality used for:
* - Changes in the actual shadow table structure (ty_shadow, avoid dumps)
* - Testing purpose (and that does NOT include UAT)
* - Free up the database when this interface is no longer being used
CALL FUNCTION 'POPUP_TO_CONFIRM'
EXPORTING
TEXT_QUESTION = 'Do you want to remove (selected) shadow table data ?'
IMPORTING
ANSWER = lv_answer.
if LV_ANSWER = '1'.
* Compose keys and remove ZSHADOW_INDX entries
select SRTFD from ZSHADOW_INDX into table lt_SRTFD
where relid = 'DB' and
SRTFD like 'KUNNR~%'.
* Filter out the keys that should not be deleted (align with select-options)
loop at lt_SRTFD into lv_SRTFD.
split lv_SRTFD at '~' into lv_SRTFD_fields-dummy lv_SRTFD_fields-kunnr.
if not ( lv_SRTFD_fields-kunnr in gr_kunnr ).
delete lt_SRTFD.
endif.
endloop.
if lt_SRTFD[] is initial.
message 'No data removed (no matches)' type 'S'.
else.
loop at lt_SRTFD into lv_SRTFD.
delete from database ZSHADOW_INDX(DB) id lv_SRTFD.
endloop.
message 'Shadow table data removed' type 'S'.
endif.
endif.
endmethod.
method read.
data: lv_SRTFD type INDX_SRTFD.
clear: gw_shadow, gw_shadow-coco[].
concatenate 'KUNNR' kunnr into lv_SRTFD separated by '~'.
if gv_CHECK = abap_true.
select SINGLE CREATE_DATE CREATE_TIME CHANGE_DATE CHANGE_TIME PROGNAME
from ZSHADOW_INDX into CORRESPONDING FIELDS OF gw_shadow_indx
where relid = 'DB' and
SRTFD = lv_SRTFD and
SRTF2 = 0.
if sy-subrc = 0.
if not gv_RESEND_DATE is initial.
if gv_RESEND_DATE < gw_shadow_indx-CHANGE_DATE.
import gw_shadow = gw_shadow from database ZSHADOW_INDX(DB) id lv_SRTFD.
endif.
else.
import gw_shadow = gw_shadow from database ZSHADOW_INDX(DB) id lv_SRTFD.
endif.
else.
clear gw_shadow_indx.
endif.
endif.
gw_shadow-kunnr = kunnr.
endmethod.
method save.
data: lv_SRTFD type INDX_SRTFD.
if gv_UPDATE = abap_true.
if gw_shadow_indx-CREATE_DATE is initial.
gw_shadow_indx-CREATE_DATE = sy-datum.
gw_shadow_indx-CREATE_TIME = sy-uzeit.
gw_shadow_indx-PROGNAME = sy-repid.
else.
gw_shadow_indx-CHANGE_DATE = sy-datum.
gw_shadow_indx-CHANGE_TIME = sy-uzeit.
endif.
concatenate 'KUNNR' gw_shadow-kunnr into lv_SRTFD separated by '~'.
MOVE-CORRESPONDING gw_shadow_indx to ZSHADOW_INDX.
export gw_shadow = gw_shadow to database ZSHADOW_INDX(DB) id lv_SRTFD.
if sy-subrc <> 0.
lcl_logging=>set_error( message = 'Schaduw tabel bijwerken: fout opgetreden (&)' par1 = lv_SRTFD ).
endif.
endif.
endmethod.
ENDCLASS.
Step 5. Define the structure of your shadow
With a type definition the relevant information on your interface can be captured. If it's a file with textual lines, a string for each of these lines would be just fine. But feel free to capture your record as fields or structures. The example below caters for a header line with debtor master data a an internal table with payment information.
This definition is captured in the coding above, the lcl_Shadow class definition will hold the types of the structures you want to shadow on. In the example above, the ty_masterdata and ty_coco (company code) will both be part of the ty_Shadow type which is (will be) monitored.
Step 6. Hook up screen parameters
The selection screen has parameters that the lcl_Shadow class will need to know about. Simply add the following coding after the START-OF-SELECTION of your interface.
if not pa_share is initial and pa_shaup = abap_true.
message 'Resend date: update not allowed' type 'S'.
elseif not pa_share is initial and pa_shaco = abap_false.
message 'Resend date: check is mandatory' type 'S'.
else.
lcl_shadow=>gv_CHECK = pa_shaco.
lcl_shadow=>gv_UPDATE = pa_shaup.
lcl_shadow=>gv_RESEND_DATE = pa_share.
* ... any other coding here
endif.
Step 7. Check your changes
The final step is performing the check. Find the location where you are "writing" the file (or calling PI functionality or calling an RFC module or creating your Idoc or whichever way the interface produces it's output) and test the output results against the shadow image. The example controls whether a simple text line is added to our .CSV file - or not.
* Shadow checks: read the information that has been sent out before:
lcl_shadow=>read( kunnr = lw_kna1-kunnr ).
if lcl_shadow=>gw_shadow-masterdata-content <> lv_line_on_file.
*-------------------------------------------------------------------------
transfer lv_line_on_file to filename.
*-------------------------------------------------------------------------
lcl_shadow=>gw_shadow-masterdata-content = lv_line_on_file.
lcl_shadow=>save( ).
add 1 to lv_written.
endif.
That's it and that's all. Unless you want to see a read-to-run report that demonstrates the setup ? You'll need to create the table described in step 2 first !
.
An the downloadable report above: there is a logical file path on the example interface report, which is not very likely to exist on your system. Create is or adjust the logical file name according to your wished. The interface used Business Application Logging which makes keeping track of any messages very easy, and if you would choose so, adding the line lcl_logging=>go_log->save( ). would store the log for future reference through transaction SLG1.