I have opened a Case with SolarWinds.
Based on another issue with displaying traps with Account Limitations and using a custom property as the limitation type. I looked into our DB and using the sp_whoisactive discovered the Top X traps query is never returning.
Here is the stock SQL Query used in the SolarWinds code. I added the specific nodeid and start and end time as those were variables in the query.
SELECT Traps.*, NodesData.NodeID AS LinkNodeID, NodesData.Caption, NodesData.IP_Address, TrapVarbinds.OID, TrapVarbinds.OIDValue, TrapVarbinds.OIDName
FROM TrapVarbinds
LEFT OUTER JOIN Traps ON TrapVarbinds.TrapID = Traps.TrapID
LEFT OUTER JOIN NodesData On Traps.NodeID = NodesData.NodeID
LEFT OUTER JOIN NodesCustomProperties (NOLOCK) ON NodesData.NodeID = NodesCustomProperties.NodeID
WHERE (((((NodesCustomProperties.CustomerOwner LIKE 'Cust1')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust2')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust3')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust4')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust5')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust6')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust7')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust8')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust9')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust10')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust11')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust12')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust13')))))
AND TrapVarbinds.TrapID IN
(SELECT TOP 25 Traps.TrapID from Traps
LEFT OUTER JOIN NodesData ON Traps.NodeID = NodesData.NodeID
WHERE (((((NodesCustomProperties.CustomerOwner LIKE 'Cust1')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust2')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust3')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust4')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust5')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust6')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust7')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust8')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust9')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust10')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust11')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust12')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust13')))))
AND Traps.IPAddress in (SELECT IP_Address FROM NodesData WITH (NOLOCK)
WHERE (((((NodesCustomProperties.CustomerOwner LIKE 'Cust1')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust2')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust3')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust4')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust5')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust6')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust7')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust8')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust9')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust10')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust11')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust12')
OR (NodesCustomProperties.CustomerOwner LIKE 'Cust13')))))
AND NodeID=1954) AND Traps.DateTime >= dateadd(day, -7, GetDate()) AND Traps.DateTime <= GetDate()
ORDER BY Traps.TrapID DESC)
ORDER BY Traps.DateTime Desc
This query is VERY Costly as it does full table joins with Traps
Doing these table joins before limiting the amount of data returned is a waste of resources.
LEFT OUTER JOIN Traps ON TrapVarbinds.TrapID = Traps.TrapID
LEFT OUTER JOIN NodesData On Traps.NodeID = NodesData.NodeID
LEFT OUTER JOIN NodesCustomProperties (NOLOCK) ON NodesData.NodeID = NodesCustomProperties.NodeID
These joins should be done using sub select statements that reduces the number of rows that are joined to traps and trapvarbinds.
It should also utilize the Nodes View instead of joining NodesData and NodesCustomProperties as the View already does a join and the DB speeds up the view.
Here is a reworked SQL I created that returns almost immediately using the same information and the way the limitations are used.
It only joins with the node rows that it needs to instead of every node.
SELECT Traps.*, N1.NodeID AS LinkNodeID, N1.Caption, N1.IP_Address, TrapVarbinds.OID, TrapVarbinds.OIDValue, TrapVarbinds.OIDName
FROM TrapVarbinds
LEFT OUTER JOIN Traps ON TrapVarbinds.TrapID = Traps.TrapID
LEFT OUTER JOIN (SELECT * FROM Nodes WITH (NOLOCK)
WHERE (((((CustomerOwner LIKE 'Cust1')
OR (CustomerOwner LIKE 'Cust2')
OR (CustomerOwner LIKE 'Cust3')
OR (CustomerOwner LIKE 'Cust4')
OR (CustomerOwner LIKE 'Cust5')
OR (CustomerOwner LIKE 'Cust6')
OR (CustomerOwner LIKE 'Cust7')
OR (CustomerOwner LIKE 'Cust8')
OR (CustomerOwner LIKE 'Cust9')
OR (CustomerOwner LIKE 'Cust10')
OR (CustomerOwner LIKE 'Cust11')
OR (CustomerOwner LIKE 'Cust12')
OR (CustomerOwner LIKE 'Cust13')))))
AND NodeID=1954) as N1 On Traps.NodeID = N1.NodeID
WHERE TrapVarbinds.TrapID IN
(SELECT TOP 25 Traps.TrapID from Traps
LEFT OUTER JOIN (SELECT * FROM Nodes WITH (NOLOCK)
WHERE (((((CustomerOwner LIKE 'Cust1')
OR (CustomerOwner LIKE 'Cust2')
OR (CustomerOwner LIKE 'Cust3')
OR (CustomerOwner LIKE 'Cust4')
OR (CustomerOwner LIKE 'Cust5')
OR (CustomerOwner LIKE 'Cust6')
OR (CustomerOwner LIKE 'Cust7')
OR (CustomerOwner LIKE 'Cust8')
OR (CustomerOwner LIKE 'Cust9')
OR (CustomerOwner LIKE 'Cust10')
OR (CustomerOwner LIKE 'Cust11')
OR (CustomerOwner LIKE 'Cust12')
OR (CustomerOwner LIKE 'Cust13')))))
AND NodeID=1954) as N2 ON Traps.NodeID = N2.NodeID
WHERE Traps.IPAddress in (SELECT IP_Address FROM Nodes WITH (NOLOCK)
WHERE (((((CustomerOwner LIKE 'Cust1')
OR (CustomerOwner LIKE 'Cust2')
OR (CustomerOwner LIKE 'Cust3')
OR (CustomerOwner LIKE 'Cust4')
OR (CustomerOwner LIKE 'Cust5')
OR (CustomerOwner LIKE 'Cust6')
OR (CustomerOwner LIKE 'Cust7')
OR (CustomerOwner LIKE 'Cust8')
OR (CustomerOwner LIKE 'Cust9')
OR (CustomerOwner LIKE 'Cust10')
OR (CustomerOwner LIKE 'Cust11')
OR (CustomerOwner LIKE 'Cust12')
OR (CustomerOwner LIKE 'Cust13')))))
AND NodeID=1954) AND Traps.DateTime >= dateadd(day, -7, GetDate()) AND Traps.DateTime <= GetDate()
ORDER BY Traps.TrapID DESC)
ORDER BY Traps.DateTime Desc