m.r*_*226 5 c++ algorithm performance search if-statement
有一些不同大小的IPTables(例如255或16384或512000 !!).每个表的每个条目都包含一个唯一的IP地址(十六进制格式)和一些其他值.IP总数为800万.所有IPTable的所有IP都已排序
我们需要每秒搜索IPTable 300,000次.我们当前的查找IP的算法如下:
// 10 <number of IPTables <20
//_rangeCount = number of IPTables
s_EntryItem* searchIPTable(const uint32_t & ip) {
for (int i = 0; i < _rangeCount; i++) {
if (ip > _ipTable[i].start && ip < _ipTable[i].end) {
int index = ip - _ipTable[i].start;
return (_ipTable[i].p_entry + index);
}
}
return NULL;
}
可以看出,在最坏的情况下,给定IP地址的比较数是_rangeCount*2,"if"语句检查的数量是_rangeCount.
假设我想更改searchIPTable并使用更有效的方法在IPTables中查找IP地址.据我所知,对于排序数组,二进制搜索等着名搜索算法的最佳软件实现需要log(n)比较(在最坏的情况下).
因此,查找IP地址的比较次数是log(8000000),等于~23.
问题1:
可以看出,两个算法所需的比较次数之间存在一点差距(_rangeCount vs 23),但在第一种方法中,有一些"if"语句可能会影响性能.如果你想运行第一个算法10次,显然第一个算法有更好的性能,但我知道运行两个算法3000,000次的想法!你有什么想法？
问题2:
是否有更有效的算法或解决方案来搜索IP？
好奇心激动,我写了一个测试程序(如下)并在我的macbook上运行它.
这表明基于std::unordered_map(查找时间==恒定时间)的一个终极解决方案能够以每秒560万次的800万条目搜索ip4地址表.
这很容易超出要求.
更新:回应我的批评者,我已将测试空间增加到所需的8m ip地址.我还将测试大小增加到1亿次搜索,其中20%将成为热门.
通过这个大的测试,我们可以清楚地看到与有序映射(对数时间查找)相比时使用unordered_map的性能优势.
所有测试参数都是可配置的.
#include <iostream>
#include <vector>
#include <algorithm>
#include <chrono>
#include <unordered_map>
#include <unordered_set>
#include <map>
#include <random>
#include <tuple>
#include <iomanip>
#include <utility>
namespace detail
{
template<class T>
struct has_reserve
{
template<class U> static auto test(U*p) -> decltype(p->reserve(std::declval<std::size_t>()), void(), std::true_type());
template<class U> static auto test(...) -> decltype(std::false_type());
using type = decltype(test<T>((T*)0));
};
}
template<class T>
using has_reserve = typename detail::has_reserve<T>::type;
using namespace std::literals;
struct data_associated_with_ip {};
using ip_address = std::uint32_t;
using candidate_vector = std::vector<ip_address>;
static constexpr std::size_t search_space_size = 8'000'000;
static constexpr std::size_t size_of_test = 100'000'000;
std::vector<ip_address> make_random_ip_set(std::size_t size)
{
std::unordered_set<ip_address> results;
results.reserve(size);
std::random_device rd;
std::default_random_engine eng(rd());
auto dist = std::uniform_int_distribution<ip_address>(0, 0xffffffff);
while (results.size() < size)
{
auto candidate = dist(eng);
results.emplace(candidate);
}
return { std::begin(results), std::end(results) };
}
template<class T, std::enable_if_t<not has_reserve<T>::value> * = nullptr>
void maybe_reserve(T& container, std::size_t size)
{
// nop
}
template<class T, std::enable_if_t<has_reserve<T>::value> * = nullptr>
decltype(auto) maybe_reserve(T& container, std::size_t size)
{
return container.reserve(size);
}
template<class MapType>
void build_ip_map(MapType& result, candidate_vector const& chosen)
{
maybe_reserve(result, chosen.size());
result.clear();
for (auto& ip : chosen)
{
result.emplace(ip, data_associated_with_ip{});
}
}
// build a vector of candidates to try against our map
// some percentage of the time we will select a candidate that we know is in the map
candidate_vector build_candidates(candidate_vector const& known)
{
std::random_device rd;
std::default_random_engine eng(rd());
auto ip_dist = std::uniform_int_distribution<ip_address>(0, 0xffffffff);
auto select_known = std::uniform_int_distribution<std::size_t>(0, known.size() - 1);
auto chance = std::uniform_real_distribution<double>(0, 1);
static constexpr double probability_of_hit = 0.2;
candidate_vector result;
result.reserve(size_of_test);
std::generate_n(std::back_inserter(result), size_of_test, [&]
{
if (chance(eng) < probability_of_hit)
{
return known[select_known(eng)];
}
else
{
return ip_dist(eng);
}
});
return result;
}
int main()
{
candidate_vector known_candidates = make_random_ip_set(search_space_size);
candidate_vector random_candidates = build_candidates(known_candidates);
auto run_test = [&known_candidates, &random_candidates]
(auto const& search_space)
{
std::size_t hits = 0;
auto start_time = std::chrono::high_resolution_clock::now();
for (auto& candidate : random_candidates)
{
auto ifind = search_space.find(candidate);
if (ifind != std::end(search_space))
{
++hits;
}
}
auto stop_time = std::chrono::high_resolution_clock::now();
using fns = std::chrono::duration<long double, std::chrono::nanoseconds::period>;
using fs = std::chrono::duration<long double, std::chrono::seconds::period>;
auto interval = fns(stop_time - start_time);
auto time_per_hit = interval / random_candidates.size();
auto hits_per_sec = fs(1.0) / time_per_hit;
std::cout << "ip addresses in table: " << search_space.size() << std::endl;
std::cout << "ip addresses searched: " << random_candidates.size() << std::endl;
std::cout << "total search hits : " << hits << std::endl;
std::cout << "searches per second : " << std::fixed << hits_per_sec << std::endl;
};
{
std::cout << "building unordered map:" << std::endl;
std::unordered_map<ip_address, data_associated_with_ip> um;
build_ip_map(um, known_candidates);
std::cout << "testing with unordered map:" << std::endl;
run_test(um);
}
{
std::cout << "\nbuilding ordered map :" << std::endl;
std::map<ip_address, data_associated_with_ip> m;
build_ip_map(m, known_candidates);
std::cout << "testing with ordered map :" << std::endl;
run_test(m);
}
}
示例结果:
building unordered map:
testing with unordered map:
ip addresses in table: 8000000
ip addresses searched: 100000000
total search hits : 21681856
searches per second : 5602458.505577
building ordered map :
testing with ordered map :
ip addresses in table: 8000000
ip addresses searched: 100000000
total search hits : 21681856
searches per second : 836123.513710
测试条件:
MacBook Pro (Retina, 15-inch, Mid 2015)
Processor: 2.2 GHz Intel Core i7
Memory: 16 GB 1600 MHz DDR3
Release build (-O2)
在主电源上运行.