Best practise summary
Perhaps each ERP implementation is connected with the need to plan and perform operations in batch mode. Regular booking of goods or invoicing of sales are some examples. But the batch processing will be used also by end users for one-off activation of operations with high demands on computation performance – e.g. elaboration of a report or start of balances at the end of fiscal periods.
Microsoft Dynamics AX has been supporting batch processing since its early versions. Each new issue always brings a number of important changes – while Axapta 3.0 processed the batch jobs with the help of a reserved client, AX 2009 brought the concept of server processing. AX 2012 brought another distinctive performance rise thanks to the shift of the batch processing to the .NET framework world.
We at Blue Dynamic have rich experience with operation of batch jobs in different environments of different customers. We want to use the following text to share, in form of five tips, some valuable knowledge and experience we have got in the course of our work. The following text will refer to the Blue Dynamic Batch pack with application modifications we have developed at solving problems related to the batch operation.
Tip 0: Some theory cannot hurt you
In other words, this is not a tip yet. But for easier writing, we should make a clear idea of the terminology. Each batch job consists of a heading (Batch job) and one or more lines (Batch task).
Each batch job consists of a heading (Batch job) and one or more lines (Batch task).
It is important to know what the Batch jobs and the Batch tasks are used for and how the Microsoft Dynamics AX approaches them. Batch jobs define the behaviour of the batch as a whole – the current status, the time schedule of processing including repetition, date and time of next run. Batch tasks, on the other hand, constitute the specific steps in batch processing – they define which actions are to be carried out and with which parameters (e.g. input filters). Dependences can be defined between individual steps, thus specifying the correct processing sequence or allowing parallel processing.
Tip 1: Good names are essential
If operating five batches in Dynamics AX , their names don’t matter too much; we can easily keep track of the batches running in the system. But in the course of time, the situation usually starts becoming more complicated. Some batches are to run only during the day, other ones only during the night. We wish to activate some batches each couple of minutes in the mode of “processing only changes” and then, as a precaution, once a day in the mode of “processing everything” (e.g. transmission of retail master data from the headquarters to individual shops is a good example). Some batches are so time consuming that we decide to relieve the system a little and process the data extents one by one (e.g. when searching duplications in a list containing a million of customers, we start with those whose surname starts with letter A). And so we see, sooner or later, that the list of active batches is several screens long.
It is therefore reasonable to introduce a system in the chaos just from the beginning. In our work, the designation of batches with names containing different prefixes has proved successful. The prefixes even can have several levels. We state the example of two-level names, distinguished by:
- Whether they run during the day (D) or at night (N)
- What area (module) they concern – e.g. logistics (LOG), finance (FIN), administration (ADM), etc.
The batch can be called e.g. N-FIN-Update balances for dimension set Account.
The advantages of such system are obvious – when arranging the batches by name, the topic-related batches will land close to each other. Additionally, we will easily distinguish batches that are duly planned and preset from batches activated ad hoc by a common user. Last but not least, the batch name can reveal information about its factual contents and run schedule.
Perfect use prefix
Tip 2: Administrator on holiday
Suppose that we have a Batch job with thirty Batch tasks, each of them with a preset complicated filter. On top of that, complicated dependences are defined among individual Batch tasks. After some time, the demand of adding one more Batch task comes. Administrator Peter who defined the original Batch job is on holiday, and so Paul is charged to solve the demand. Paul gradually reveals that to complete the task, he must be logged on as administrator and that the status of the heading must be Withhold. However, finally he sees that he cannot complete the task anyway – when establishing a new record in Batch tasks, he receives a quite uninformative error message “Cannot create a record in Batch transactions (Batch)”. The corresponding AOS validation failed. He would succeed, or rather fail, equally if he only wanted to change dependences among the existing Batch tasks. And what now?
To understand what happens in this case, we must know how Dynamics AX classifies a logged-in user in relation to changes of definition of batches. The logged-in user belongs to one of the following three categories:
- batch owner, i.e. the user who created the original batch definition
- system administrator
- neither of the above stated users
The first and the third case are two extremes. The batch owner can handle his/her batch at will. On the contrary, a common user without administrator authorizations cannot do anything with another user’s batch. Interesting is case number two – the administrator who is not the author of the batch. He/she can change the data concerning the actual execution of the batch – i.e. suspend it or plan it according to a new schedule. But he/she cannot change the factual content of the batch – he/she cannot create new Batch tasks, change the parameters (e.g. input filters) of the existing Batch tasks records or change the actual action (i.e. the name of class). The motive for such restrictions is to allow an audit of changes – it is not admissible that Paul carries out changes of the batch settings leading to undesirable behaviour, all that on behalf of Peter. So Paul has two options now – either get Peter or create his own definition of a Batch job just from the beginning.
To avoid breaching of the audit principles while allowing administrator Peter to leave for holiday, we have developed a new functionality in the Blue Dynamic Batch package. Instead of the value of the field “Created by”, we use a new field “Batch job owner” for the needs of the audit. After establishing the batch, the value of the field is the same as the value in the field “Created by”, but it can be changed with the help of the „Function -> Take over ownership“.
After taking over the ownership, Paul can carry out the required changes, of course on his own behalf. The value “Batch job owner” is written through also to the history of batch processing, so we can easily find out when the ownership was changed.
Tip 3: Purposes and limitations of batch groups
Let’s start by looking at the limitations of the batch groups – they are not useful to organize work with individual Batch jobs. The reason is simple – the batch group is a feature of the Batch task, not Batch job record. If the need emerges to better organize the Batch jobs, order must be introduced into the batch designation, see Tip 1.
So what is the actual purpose of the batch groups? The main purpose consists in the opportunity to process different Batch tasks at different AOS servers. In installations where the Batch jobs are processed by one AOS server only, a single batch group will be sufficient for us (a group without name is automatically created for such purposes after the installation of AX). But situations do exist where the division in individual AOSs can come in handy. Examples can include batches with high demands on computation performance or batches requiring a special source, available at some AOS servers only.
We at Blue Dynamic considered how the batch groups could serve even better. One of the problems often faced at batch operation consists in mutual negative effects of two or more batches running in parallel. Sometimes, the “Post inventory” retail batch meets the “Post statement” retail batch. The run of “Post inventory” is planned for each 10 minutes, while “Post statement” should run once a day after the cash registers were closed. When the cash registers are closed, the “Post inventory” batch actually performs an “idle run” – the logic of the matter implies that no new transactions will arrive. So if the “Post inventory” could be preset to run until a specific hour only, and then start up only the next day in the morning, we could easily avoid such collision and get rid of complicated ascertaining how the two batches influence each other and how they could be rewritten to take the parallel run into account.
The above stated problem can be solved in standard AX only by using multiple application servers. We create two batch groups – DAY and NIGHT. We include the “Post inventory” batch in the DAY group and the “Post statement” batch in the NIGHT group. We set the first AOS to run from 8:00 to 22:00 and to process only the DAY batch group. We set the second AOS to run from 22:15 to 7:45 (allowing for a 15-minute reserve for rundown of batches taking more time) and to process only the NIGHT batch group.
But what can we do if we cannot use any additional AOS server? We will solve the problem by adding the option of entering a schedule at the level of batch groups.
After this modification, included in the Blue Dynamic Batch package, a single AOS will be sufficient and no conflict will emerge.
Tip 4: “Self-proliferation” of batches and “nameless” batch group
In this tip, we will go back to our “Post statement” batch from the preceding tip. The customer operates a shop network and uses the AX Retail module. During the day, he sells and records the takings in POS. In the evening, the cash registers are balanced and the data are transmitted to AX. A report is then created in AX and it should be posted. But as we want to execute everything automatically, the post statement job is planned as a night job. So just after planning, we have a Batch job called “Post statement” that has one Batch task with the same name and with the NIGHT batch group. But after the batch starts to be processed, new Batch tasks, not existing before, suddenly emerge. What are these tasks?
The thing is that some batch jobs are written so that they divide the total work into smaller units and process those units in parallel in multiple Batch tasks. Such dynamic Batch tasks may even have mutual dependences defined – e.g. a final Batch task, charged with cleaning activities that can take place only after all Batch tasks have been completed, can be placed at the end.
This nice functionality has a tricky aspect. The thing is that dynamically created Batch tasks might not have the same batch group as the original Batch task; the exact behaviour is defined by the programmer when developing the batch. The result is obvious – if the dynamically created Batch tasks have a group assigned and the execution of such group is suspended, they will never be started and the batch as a whole will be left “hanging” in the Processing status.
Such situation is illustrated by the following screen – the last three Batch tasks have a nameless batch group assigned. If the processing of a nameless batch group is suspended, the posting of the report will never be finished.
It is actually a typical case – the dynamically created Batch tasks will probably have the same batch group as the main Batch task or it will be empty. As the inner logic of creation of dynamic Batch tasks cannot always be influenced, it is a good idea not to restrict the processing of nameless batch groups in any way. That is important for one more reason – one-off batches activated by ad hoc end users will have a nameless batch group very often, as less experienced users often forget to mention the batch group.
Tip 5: How to secure restful sleep
Night hours are often intended to run batches whose successful completion conditions correct operation of the company on the next day – they may include for example the master plan generating proposals of orders and restocking of shops or a batch preparing work for issue from central warehouse.
Batches take a lot of time, so there might not be enough time for correction next day in the morning.In general, these are relatively complicated tasks facing a lot of obstacles that might prevent them from successful completion. On top of that, such batches take a lot of time, so there might not be enough time for correction next day in the morning. It is therefore very important to respond quickly to problems related to the run of such batches.
The standard AX offers the use of alerts in this respect. AX can be asked to create an alert for each batch if one of the following events occur:
- the batch is duly completed
- the batch ends with an error
- the batch is cancelled
The administrator can view the alerts in AX application or have them sent by e-mail to a predefined address.
The alert functionality is undoubtedly useful, but it will not be enough to secure restful sleep. A problem will emerge if the processing of the batch does not start at all, or if its processing takes too much time. A frequent cause of prolonged duration may consist in negative interaction with another job with high demands on computation performance.
AX is often not the only Company system involved in the success of the next day. Companies often operate independent CRM, e-shop, POS systems, etc. In such cases, the demands on supervision over the whole ecosystem rise, and the infrastructure should be monitored from one central place. In such case, alerts will constitute a non-systemic solution and the AX batches should be connected to central monitoring instead.
Blue Dynamic successfully uses the Check MK system in connection with batch monitoring. We have developed scripts that allow to monitor the run of critical batches – to see whether the processing started at the defined time, how does it progress and whether they will be completed by the prescribed time – and to respond to critical situations by sending an SMS to the support phone. We are working at generalization and at the opportunity to enter monitoring-related parameters directly in the AX environment.
Ondřej Liberda works as a Dynamics AX Developer/Architect in the company Blue Dynamic and has extensive experience with Dynamics AX development.