fig. 1 We start from the base, aim and up vector.
fig. 2 We are now finding the (up - base) vector and the new aim, which is essentially the vector we are going to use to aim the object.
fig. 3 Doing the cross product between the new aim and the (up - base) allow us to find the normal vector, which will be used as a side vector for our aim constraint. Notice that the cross product points inward/outward depending on the right hand rule. It is a good idea to normalize our cross product (dividing it by its magnitude to get the same vector but with the magnitude of the unit vector).
fig. 4 Doing now the cross product between the side and the new aim (and normalizing) will give us a perpendicular up vector.
fig. 5 We now have 3 perpendicular vectors (it may be a good idea to normalize the new aim as well) and we can compose a 4 by 4 matrix. The last row represents the transform and since in an aim constraint that doesn't need to change we can set it back to the obj's base coordinates.
What if we want to add an offset to this aim constraint? This is extremely easy. When we apply an aim constraint with the "mantain offset" option turned on, for instance in Maya, it basically just stores the three base rotation values of the object to a 3 float attribute and adds them to the result rotation of the decomposed matrix.