Request Validation and Filtering by Flags – Redesign and Refactoring

General
In the previous posts I started describing a validation / filtering framework we’re building.
While showing the code, I am trying to show clean code, test orientation and code evolution.
It has some agility in the process; We know the end requirements, but the exact details are evolving over time.

During the development we have changed the code to be more general as we saw some patterns in it.
The code evolved as the flow evolved as well.

The flow as we now understand it
Here’s a diagram of the flow we’ll implement

Request Sequence

Request Sequence

The Pattern
At each step of the sequence (validation, filtering, action), we recognized the same pattern:

  1. We have specific implementations (filters, validations)
  2. We have an engine that wraps up the specific implementations
  3. We need to map the implementations by flag, and upon request’s flags, select the appropriate implementations.
  4. We need to have a class that calls the mapper and then the engine

A diagram showing the pattern

The Pattern

The Pattern

Source Code
In order to show some of the evolution of the code, and how refactoring changed it, I added tags in GitHub after major changes.

Code Examples
Let’s see what came up from the mapper pattern.

public interface MapperByFlag<T> {
  List<T> getOperations(Request request);
}
public abstract class AbstractMapperByFlag<T> implements MapperByFlag<T> {
  private List<T> defaultOperations;
  private Map<String, List<T>> mapOfOperations;

  public AbstractMapperByFlag(List<T> defaultOperations, Map<String, List<T>> mapOfOperations) {
    this.defaultOperations = defaultOperations;
    this.mapOfOperations = mapOfOperations;
  }

  @Override
  public final List<T> getOperations(Request request) {
    Set<T> selectedFilters = Sets.newHashSet(defaultOperations);
    Set<String> flags = request.getFlags();
    for (String flag : flags) {
      if (mapOfOperations.containsKey(flag)) {
        selectedFilters.addAll(mapOfOperations.get(flag));
      }
    }
    return Lists.newArrayList(selectedFilters);
  }
}
  public RequestValidationByFlagMapper(List<RequestValidation> defaultValidations,
    map<String, List<RequestValidation>> mapOfValidations) {
    super(defaultValidations, mapOfValidations);
  }

  public ItemFiltersByFlagMapper(List<Filter> defaultFilters, Map<String, List<Filter>> mapOfFilters) {
    super(defaultFilters, mapOfFilters);
  }

I created a test for the abstract class, to show the flow itself.
The tests of the implementations use Java Reflection to verify that the correct injected parameters are sent to the super.
I am showing the imports here as well. To have some reference for the static imports, mockito and hamcrest packages and classes.

import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.junit.Assert.assertThat;
import static org.mockito.Mockito.when;

import java.util.List;
import java.util.Map;

import org.eyal.requestvalidation.model.Request;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.Mock;
import org.mockito.runners.MockitoJUnitRunner;

import com.google.common.collect.ImmutableMap;
import com.google.common.collect.Lists;
import com.google.common.collect.Sets;

@RunWith(MockitoJUnitRunner.class)
public class AbstractMapperByFlagTest {
	private final static String FLAG_1 = "flag 1";
	private final static String FLAG_2 = "flag 2";

	@Mock
	private Request request;

	private String defaultOperation1 = "defaultOperation1";
	private String defaultOperation2 = "defaultOperation2";
	private String mapOperation11 = "mapOperation11";
	private String mapOperation12 = "mapOperation12";
	private String mapOperation23 = "mapOperation23";

	private MapperByFlag<String> mapper;

	@Before
	public void setup() {
		List<String> defaults = Lists.newArrayList(defaultOperation1, defaultOperation2);
		Map<String, List<String>> mapped = ImmutableMap.<String, List<String>> builder()
		        .put(FLAG_1, Lists.newArrayList(mapOperation11, mapOperation12))
		        .put(FLAG_2, Lists.newArrayList(mapOperation23, mapOperation11)).build();
		mapper = new AbstractMapperByFlag<String>(defaults, mapped) {
		};
	}

	@Test
	public void whenRequestDoesNotHaveFlagsShouldReturnDefaultFiltersOnly() {
		when(request.getFlags()).thenReturn(Sets.<String> newHashSet());

		List<String> filters = mapper.getOperations(request);
		assertThat(filters, containsInAnyOrder(defaultOperation1, defaultOperation2));
	}

	@Test
	public void whenRequestHasFlagsNotInMappingShouldReturnDefaultFiltersOnly() {
		when(request.getFlags()).thenReturn(Sets.<String> newHashSet("un-mapped-flag"));
		List<String> filters = mapper.getOperations(request);
		assertThat(filters, containsInAnyOrder(defaultOperation1, defaultOperation2));
	}
	
	@Test
	public void whenRequestHasOneFlagShouldReturnWithDefaultAndMappedFilters() {
		when(request.getFlags()).thenReturn(Sets.<String> newHashSet(FLAG_1));
		List<String> filters = mapper.getOperations(request);
		assertThat(filters, containsInAnyOrder(mapOperation12, defaultOperation1, mapOperation11, defaultOperation2));
	}
	
	@Test
	public void whenRequestHasTwoFlagsShouldReturnWithDefaultAndMappedFiltersWithoutDuplications() {
		when(request.getFlags()).thenReturn(Sets.<String> newHashSet(FLAG_1, FLAG_2));
		List<String> filters = mapper.getOperations(request);
		assertThat(filters, containsInAnyOrder(mapOperation12, defaultOperation1, mapOperation11, defaultOperation2, mapOperation23));
	}
}
@RunWith(MockitoJUnitRunner.class)
public class RequestValidationByFlagMapperTest {

	@Mock
	private List<RequestValidation> defaultValidations;
    
	@Mock
	private Map<String, List<RequestValidation>> mapOfValidations;

	@InjectMocks
	private RequestValidationByFlagMapper mapper;

	@SuppressWarnings("unchecked")
    @Test
	public void verifyParameters() throws NoSuchFieldException, SecurityException, IllegalArgumentException,
	        IllegalAccessException {
		Field defaultOperationsField = AbstractMapperByFlag.class.getDeclaredField("defaultOperations");
		defaultOperationsField.setAccessible(true);
        List<RequestValidation> actualFilters = (List<RequestValidation>) defaultOperationsField.get(mapper);
		assertThat(actualFilters, sameInstance(defaultValidations));

		Field mapOfFiltersField = AbstractMapperByFlag.class.getDeclaredField("mapOfOperations");
		mapOfFiltersField.setAccessible(true);
		Map<String, List<RequestValidation>> actualMapOfFilters = (Map<String, List<RequestValidation>>) mapOfFiltersField.get(mapper);
		assertThat(actualMapOfFilters, sameInstance(mapOfValidations));
	}
}

To Do
There are other classes that might be candidate for refactoring of some sort.
RequestFlowValidation and RequestFilter are similar.
And
RequestValidationsEngineImpl and FiltersEngine

To Do 2
Create a Matcher for the reflection part.

Code
As always, all the code can be found at:

A Tag for this post: all-components-in

Conclusion
The infrastructure is almost done.
During this time we are also implementing actual classes for the flow (validations, filters, actions).
These are not covered in the posts, nor in GitHub.
The infrastructure will be wired to a service we have using Spring.
This will be explained in future posts.

Advertisement

Request Validation and Filtering by Flags – Filtering an Item

On a previous post, I introduced a system requirement of validating and filtering a request by setting flags on it.

Reference: Introduction

In this post I want to show the filtering system.

Here are general UML diagrams of the filtering components and sequence.

Filtering UML Diagram

General Components

public interface Item {
        String getName();
}
public interface Request {
        Set getFlags();
        List getItems();
}

Filter Mechanism (as described in the UML above)

public interface Filter extends Predicate {
	String errorMessage();
}

FilterEngine is a cool part, which takes several Filters and apply to each the items. Below you can see the code of it. Above, the sequence diagram shows how it’s done.

public class FiltersEngine {

	public FiltersEngine() {
	}

	public ItemsFilterResponse applyFilters(List filters, List items) {
		List validItems = Lists.newLinkedList(items);
		List invalidItemInformations = Lists.newLinkedList();
		for (Filter validator : filters) {
			ItemsFilterResponse responseFromFilter = responseFromFilter(validItems, validator);
			validItems = responseFromFilter.getValidItems();
			invalidItemInformations.addAll(responseFromFilter.getInvalidItemsInformations());
		}

		return new ItemsFilterResponse(validItems, invalidItemInformations);
	}

	private ItemsFilterResponse responseFromFilter(List items, Filter filter) {
		List validItems = Lists.newLinkedList();
		List invalidItemInformations = Lists.newLinkedList();
		for (Item item : items) {
			if (filter.apply(item)) {
				validItems.add(item);
			} else {
				invalidItemInformations.add(new InvalidItemInformation(item, filter.errorMessage()));
			}
		}
		return new ItemsFilterResponse(validItems, invalidItemInformations);
	}
}

And of course, we need to test it:

@RunWith(MockitoJUnitRunner.class)
public class FiltersEngineTest {
	private final static String MESSAGE_FOR_FILTER_1 = "FILTER - 1 - ERROR";
	private final static String MESSAGE_FOR_Filter_2 = "FILTER - 2 - ERROR";
	@Mock(name = "filter 1")
	private Filter singleFilter1;
	@Mock(name = "filter 2")
	private Filter singleFilter2;
	@Mock(name = "item 1")
	private Item item1;
	@Mock(name = "item 2")
	private Item item2;

	@InjectMocks
	private FiltersEngine filtersEngine;

	@Before
	public void setup() {
		when(singleFilter1.errorMessage()).thenReturn(MESSAGE_FOR_FILTER_1);
		when(singleFilter2.errorMessage()).thenReturn(MESSAGE_FOR_Filter_2);

		when(item1.getName()).thenReturn("name1");

		when(item2.getName()).thenReturn("name2");
	}

	@Test
	public void verifyThatAllSingleFiltersAreCalledForValidItems() {
		when(singleFilter1.apply(item1)).thenReturn(true);
		when(singleFilter1.apply(item2)).thenReturn(true);
		when(singleFilter2.apply(item1)).thenReturn(true);
		when(singleFilter2.apply(item2)).thenReturn(true);

		ItemsFilterResponse response = filtersEngine.applyFilters(Lists.newArrayList(singleFilter1, singleFilter2),
				Lists.newArrayList(item1, item2));
		assertThat("expected no invalid", response.getInvalidItemsInformations(),
				emptyCollectionOf(InvalidItemInformation.class));
		assertThat(response.getValidItems(), containsInAnyOrder(item1, item2));

		verify(singleFilter1).apply(item1);
		verify(singleFilter1).apply(item2);
		verify(singleFilter2).apply(item1);
		verify(singleFilter2).apply(item2);
		verifyNoMoreInteractions(singleFilter1, singleFilter2);
	}

	@SuppressWarnings("unchecked")
	@Test
	public void itemsFailIndifferentFiltersShouldGetOnlyFailures() {
		when(singleFilter1.apply(item1)).thenReturn(false);
		when(singleFilter1.apply(item2)).thenReturn(true);
		when(singleFilter2.apply(item2)).thenReturn(false);

		ItemsFilterResponse response = filtersEngine.applyFilters(Lists.newArrayList(singleFilter1, singleFilter2),
				Lists.newArrayList(item1, item2));
		assertThat(
				response.getInvalidItemsInformations(),
				containsInAnyOrder(matchInvalidInformation(new InvalidItemInformation(item1, MESSAGE_FOR_FILTER_1)),
						matchInvalidInformation(new InvalidItemInformation(item2, MESSAGE_FOR_Filter_2))));
		assertThat(response.getValidItems(), emptyCollectionOf(Item.class));

		verify(singleFilter1).apply(item1);
		verify(singleFilter1).apply(item2);
		verify(singleFilter1).errorMessage();
		verify(singleFilter2).apply(item2);
		verify(singleFilter2).errorMessage();
		verifyNoMoreInteractions(singleFilter1, singleFilter2);
	}

	@Test
	public void firstItemFailSecondItemSuccessShouldGetOneItemInEachList() {
		when(singleFilter1.apply(item1)).thenReturn(true);
		when(singleFilter1.apply(item2)).thenReturn(true);
		when(singleFilter2.apply(item1)).thenReturn(false);
		when(singleFilter2.apply(item2)).thenReturn(true);

		ItemsFilterResponse response = filtersEngine.applyFilters(Lists.newArrayList(singleFilter1, singleFilter2),
				Lists.newArrayList(item1, item2));
		assertThat(response.getInvalidItemsInformations(), contains(matchInvalidInformation(new InvalidItemInformation(item1,
				MESSAGE_FOR_Filter_2))));
		assertThat(response.getValidItems(), containsInAnyOrder(item2));

		verify(singleFilter1).apply(item1);
		verify(singleFilter1).apply(item2);
		verify(singleFilter2).apply(item1);
		verify(singleFilter2).apply(item2);
		verify(singleFilter2).errorMessage();
		verifyNoMoreInteractions(singleFilter1, singleFilter2);
	}

	private static BaseMatcher matchInvalidInformation(InvalidItemInformation expected) {
		return new InvalidItemInformationMatcher(expected);
	}

	private final static class InvalidItemInformationMatcher extends BaseMatcher {
		private InvalidItemInformation expected;

		private InvalidItemInformationMatcher(InvalidItemInformation expected) {
			this.expected = expected;
		}

		public boolean matches(Object itemInformation) {
			InvalidItemInformation actual = (InvalidItemInformation) itemInformation;
			return actual.getName().equals(expected.getName())
					&& actual.getErrorMessage().equals(expected.getErrorMessage());
		}

		public void describeTo(Description description) {
		}
	}
}

Some explanation about the test
You can see that I don’t care about the implementation of Filter. Actually, I don’t even have any implementation of it.
I also don’t have implementation of the Item nor the request.
You can see an example of how to create a BaseMatcher to be used with assertThat(…)

Coding
Try to see whether it is ‘clean’. Can you understand the story of the code? Can you tell what the code does by reading it line by line?

On the following post I will show how I applied the flag mapping to select the correct filters for a request.

You can find all the code in: https://github.com/eyalgo/request-validation

[Edit] Created tag Filtering_an_item before refactoring.

Request Validation and Filtering by Flags – Introduction

General

We are working on a service that should accept some kind of request.

The request has List of Items. In the response we need to tell the client whether the request is valid and also some information about each item: is it valid or not. If it’s valid, it will be persisted. If it’s not, it should be filtered out. So the response can have information of how many items are valid (and sent to be persisted) and list of information of the filtered out items.

The request has another metadata in it. It has collection (set) of flags. The filtering and validation is based on the flags of the request. So basically one request may be validated and filtered differently than the other, based on the flags of each request.

We might have general validations / filters that need to be applied to any request, whatever flags it has.

Request Validation and Filtering High level design

Design

Flags Mapping

We’ll hold a mapping of flag-to-filters, and flag-to-validation.

Request

Has flags and items.

Components

Filter, Filter-Engine, Flags-Mapper

Development Approach

Bottom Up

We have a basic request already, as the service is up and running, but we don’t have any infrastructure for flags, flag-mapping, validation and filtering.

We’ll work bottom up; create the mechanism for filtering, enhance the request and then wire it up using Spring.

Coding

I’ll try to show the code using tests, and the development using some kind of TDD approach.

I am using eclipse’s EclEmma for coverage.

General

By looking at the code, you can see usage of JUnit, Mockito, Hamcrest, Google-Guava.

You can also see small classes, and interface development approach.

Source Code

https://github.com/eyalgo/request-validation